Background For many years it has been claimed that observationalstudies find stronger treatment effects than randomized, controlledtrials. We compared the results of observational studies withthose of randomized, controlled trials.
Methods We searched the Abridged Index Medicus and Cochranedata bases to identify observational studies reported between1985 and 1998 that compared two or more treatments or interventionsfor the same condition. We then searched the Medline and Cochranedata bases to identify all the randomized, controlled trialsand observational studies comparing the same treatments forthese conditions. For each treatment, the magnitudes of theeffects in the various observational studies were combined bythe Mantel–Haenszel or weighted analysis-of-variance procedureand then compared with the combined magnitude of the effectsin the randomized, controlled trials that evaluated the sametreatment.
Results There were 136 reports about 19 diverse treatments,such as calcium-channel–blocker therapy for coronary arterydisease, appendectomy, and interventions for subfertility. Inmost cases, the estimates of the treatment effects from observationalstudies and randomized, controlled trials were similar. In only2 of the 19 analyses of treatment effects did the combined magnitudeof the effect in observational studies lie outside the 95 percentconfidence interval for the combined magnitude in the randomized,controlled trials.
Conclusions We found little evidence that estimates of treatmenteffects in observational studies reported after 1984 are eitherconsistently larger than or qualitatively different from thoseobtained in randomized, controlled trials.
Source Information
From the Department of Family Medicine, University of Iowa College of Medicine, Iowa City.
Address reprint requests to Dr. Hartz at the Department of Family Medicine, University of Iowa College of Medicine, 01292-D PFP, Iowa City, IA 52242-1097, or at arthur-hartz{at}uiowa.edu.
Hippisley-Cox, J., Coupland, C.
(2009). Predicting risk of osteoporotic fracture in men and women in England and Wales: prospective derivation and validation of QFractureScores. BMJ
339: b4229-b4229
[Abstract][Full Text]
Ferrer, R., Artigas, A., Suarez, D., Palencia, E., Levy, M. M., Arenzana, A., Perez, X. L., Sirvent, J.-M., for the Edusepsis Study Group,
(2009). Effectiveness of Treatments for Severe Sepsis: A Prospective, Multicenter, Observational Study. Am. J. Respir. Crit. Care Med.
180: 861-866
[Abstract][Full Text]
La Caze, A.
(2009). Evidence-Based Medicine Must Be .... J Med Philos
34: 509-527
[Abstract][Full Text]
Federman, D. G., Concato, J., Kirsner, R. S.
(2009). Screening for Skin Cancer: Absence of Evidence. Arch Dermatol
145: 926-927
[Full Text]
Bilimoria, K. Y., Bentrem, D. J., Hansen, N. M., Bethke, K. P., Rademaker, A. W., Ko, C. Y., Winchester, D. P., Winchester, D. J.
(2009). Comparison of Sentinel Lymph Node Biopsy Alone and Completion Axillary Lymph Node Dissection for Node-Positive Breast Cancer. JCO
27: 2946-2953
[Abstract][Full Text]
Doi, S. A. R., Onitilo, A.
(2009). Author Reply. Clin Med Res
7: 2-3
[Full Text]
McAllister, D, Wild, S
(2009). Why should clinicians understand epidemiology?. Postgrad. Med. J.
85: 313-315
[Abstract][Full Text]
Howick, J., Glasziou, P., Aronson, J. K
(2009). The evolution of evidence hierarchies: what can Bradford Hill's 'guidelines for causation' contribute?. JRSM
102: 186-194
[Full Text]
Schemitsch, E. H., Bhandari, M., McKee, M. D., Zdero, R., Tornetta, P. III, McGehee, J. B., Hawkins, R. J.
(2009). Orthopaedic Surgeons: Artists or Scientists?. JBJS
91: 1264-1273
[Full Text]
Van der Niepen, P., Woestenburg, A., Brie, H., Vancayzeele, S., MacDonald, K., Denhaerynck, K., Lee, C., Hermans, C., Abraham, I.
(2009). Effectiveness of Valsartan for Treatment of Hypertension: Patient Profiling and Hierarchical Modeling of Determinants and Outcomes (the PREVIEW Study). The Annals of Pharmacotherapy
43: 849-861
[Abstract][Full Text]
Bhandari, M., Morshed, S., Tornetta, P. III, Schemitsch, E. H.
(2009). Design, Conduct, and Interpretation of Nonrandomized Orthopaedic Studies: A Practical Approach. (All) Evidence Matters. JBJS
91: 1-1
[Full Text]
Hoppe, D. J., Schemitsch, E. H., Morshed, S., Tornetta, P. III, Bhandari, M.
(2009). Hierarchy of Evidence: Where Observational Studies Fit in and Why We Need Them. JBJS
91: 2-9
[Abstract][Full Text]
Li, Y., Zheng, Z., Xu, B., Zhang, S., Li, W., Gao, R., Hu, S.
(2009). Comparison of Drug-Eluting Stents and Coronary Artery Bypass Surgery for the Treatment of Multivessel Coronary Disease: Three-Year Follow-Up Results From a Single Institution. Circulation
119: 2040-2050
[Abstract][Full Text]
Hippisley-Cox, J., Coupland, C., Robson, J., Sheikh, A., Brindle, P.
(2009). Predicting risk of type 2 diabetes in England and Wales: prospective derivation and validation of QDScore. BMJ
338: b880-b880
[Abstract][Full Text]
Ziegler, R.
(2009). Mistletoe Preparation Iscador: Are there Methodological Concerns with Respect to Controlled Clinical Trials?. Evid Based Complement Alternat Med
6: 19-30
[Abstract][Full Text]
Pinnock, H, Sheikh, A
(2009). Primary care research and clinical practice: respiratory disease. Postgrad. Med. J.
85: 74-79
[Abstract][Full Text]
Tannen, R. L, Weiner, M. G, Xie, D.
(2009). Use of primary care electronic medical record database in drug efficacy research on cardiovascular outcomes: comparison of database and randomised controlled trial findings. BMJ
338: b81-b81
[Abstract][Full Text]
Srivastava, R., Downey, E. C., O'Gorman, M., Feola, P., Samore, M., Holubkov, R., Mundorff, M., James, B. C., Rosenbaum, P., Young, P. C., Dean, J. M.
(2009). Impact of Fundoplication Versus Gastrojejunal Feeding Tubes on Mortality and in Preventing Aspiration Pneumonia in Young Children With Neurologic Impairment Who Have Gastroesophageal Reflux Disease. Pediatrics
123: 338-345
[Abstract][Full Text]
Simone, C. B., Simone, C. B. II
(2008). Re: Should Supplemental Antioxidant Administration Be Avoided During Chemotherapy and Radiation Therapy?. JNCI J Natl Cancer Inst
100: 1558-1559
[Full Text]
Anderson, P. A., McCormick, P. C., Angevine, P. D.
(2008). Randomized Controlled Trials of the Treatment of Lumbar Disk Herniation: 1983-2007. J Am Acad Orthop Surg
16: 566-573
[Abstract][Full Text]
Siempos, I. I., Vardakas, K. Z., Kopterides, P., Falagas, M. E.
(2008). Adjunctive therapies for community-acquired pneumonia: a systematic review. J Antimicrob Chemother
62: 661-668
[Abstract][Full Text]
Leichsenring, F., Rabung, S.
(2008). Effectiveness of Long-term Psychodynamic Psychotherapy: A Meta-analysis. JAMA
300: 1551-1565
[Abstract][Full Text]
Makela, K. T., Eskelinen, A., Pulkkinen, P., Paavolainen, P., Remes, V.
(2008). Total Hip Arthroplasty for Primary Osteoarthritis in Patients Fifty-five Years of Age or Older. An Analysis of the Finnish Arthroplasty Registry. JBJS
90: 2160-2170
[Abstract][Full Text]
Martens, E. P, Pestman, W. R, de Boer, A., Belitser, S. V, Klungel, O. H
(2008). Systematic differences in treatment effect estimates between propensity score methods and logistic regression. Int J Epidemiol
37: 1142-1147
[Abstract][Full Text]
Boffetta, P., McLaughlin, J. K., La Vecchia, C., Tarone, R. E., Lipworth, L., Blot, W. J.
(2008). False-Positive Results in Cancer Epidemiology: A Plea for Epistemological Modesty. JNCI J Natl Cancer Inst
100: 988-995
[Abstract][Full Text]
Hannan, E. L.
(2008). Randomized Clinical Trials and Observational Studies: Guidelines for Assessing Respective Strengths and Limitations. J Am Coll Cardiol Intv
1: 211-217
[Abstract][Full Text]
Beck, E J, Mandalia, S, Youle, M, Brettle, R, Fisher, M, Gompels, M, Kinghorn, G, McCarron, B, Pozniak, A, Tang, A, Walsh, J, Williams, I, Gazzard, B
(2008). Treatment outcome and cost-effectiveness of different highly active antiretroviral therapy regimens in the UK (1996-2002). Int J STD AIDS
19: 297-304
[Abstract][Full Text]
Park, D.-W., Yun, S.-C., Lee, S.-W., Kim, Y.-H., Lee, C. W., Hong, M.-K., Kim, J.-J., Choo, S. J., Song, H., Chung, C. H., Lee, J.-W., Park, S.-W., Park, S.-J.
(2008). Long-Term Mortality After Percutaneous Coronary Intervention With Drug-Eluting Stent Implantation Versus Coronary Artery Bypass Surgery for the Treatment of Multivessel Coronary Artery Disease. Circulation
117: 2079-2086
[Abstract][Full Text]
Sanders, M. H., Montserrat, J. M., Farre, R., Givelber, R. J.
(2008). Positive Pressure Therapy: A Perspective on Evidence-based Outcomes and Methods of Application. Proc Am Thorac Soc
5: 161-172
[Abstract][Full Text]
Hemila, H.
(2008). THE AUTHOR REPLIES. Am J Epidemiol
167: 501-502
[Full Text]
Simpson, R. J Jr
(2008). Assessing the safety of drugs through observational research. Heart
94: 129-130
[Full Text]
Pagnin, D., de Queiroz, V., Pini, S., Cassano, G. B.
(2008). Efficacy of ECT in Depression: A Meta-Analytic Review. Focus
6: 155-162
[Abstract][Full Text]
Ferraris, V. A., Edwards, F. H., Shahian, D. M., Ferraris, S. P.
(2008). Risk Stratification and Comorbidity. Card Surg Adult
3: 199-246
[Full Text]
Shrier, I., Boivin, J.-F., Steele, R. J., Platt, R. W., Furlan, A., Kakuma, R., Brophy, J., Rossignol, M.
(2007). Should Meta-Analyses of Interventions Include Observational Studies in Addition to Randomized Controlled Trials? A Critical Examination of Underlying Principles. Am J Epidemiol
166: 1203-1209
[Abstract][Full Text]
Nicholl, J.
(2007). Case-mix adjustment in non-randomised observational evaluations: the constant risk fallacy. J. Epidemiol. Community Health
61: 1010-1013
[Abstract][Full Text]
Vandenbroucke, J. P., Elm, E. v., Altman, D. G., Gotzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., Egger, M., for the STROBE initiative,
(2007). Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and Elaboration. ANN INTERN MED
147: W-163-W-194
[Abstract][Full Text]
Kelly, C., Arnold, R., Galloway, Y., O'Hallahan, J.
(2007). A Prospective Study of the Effectiveness of the New Zealand Meningococcal B Vaccine. Am J Epidemiol
166: 817-823
[Abstract][Full Text]
Worrall, J.
(2007). Why There's No Cause to Randomize. Br J Philos Sci
58: 451-488
[Abstract][Full Text]
Minkovitz, C. S., Strobino, D., Mistry, K. B., Scharfstein, D. O., Grason, H., Hou, W., Ialongo, N., Guyer, B.
(2007). Healthy Steps for Young Children: Sustained Results at 5.5 Years. Pediatrics
120: e658-e668
[Abstract][Full Text]
Blumenfeld, Z.
(2007). How to Preserve Fertility in Young Women Exposed to Chemotherapy? The Role of GnRH Agonist Cotreatment in Addition to Cryopreservation of Embrya, Oocytes, or Ovaries. The Oncologist
12: 1044-1054
[Abstract][Full Text]
Shojania, K. G., Sampson, M., Ansari, M. T., Ji, J., Doucette, S., Moher, D.
(2007). How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis. ANN INTERN MED
147: 224-233
[Abstract][Full Text]
Auerbach, A. D., Landefeld, C. S., Shojania, K. G.
(2007). The Tension between Needing to Improve Care and Knowing How to Do It. NEJM
357: 608-613
[Full Text]
Berry, D. A., Ravdin, P. M.
(2007). Breast Cancer Trends: A Marriage Between Clinical Trial Evidence and Epidemiology. JNCI J Natl Cancer Inst
99: 1139-1141
[Full Text]
Hodgson, R., Bushe, C., Hunter, R.
(2007). Measurement of long-term outcomes in observational and randomised controlled trials. Br. J. Psychiatry
191: s78-s84
[Abstract][Full Text]
Maeng, M., Okkels Jensen, L., Rasmussen, K., Flensted Lassen, J., Romer Krusell, L., Thayssen, P., Thuesen, L.
(2007). Target lesion revascularisation in patients treated with a sirolimus-eluting or paclitaxel-eluting stent. Heart
93: 694-697
[Abstract][Full Text]
The Society of Thoracic Surgeons Blood Conservatio, , Ferraris, V. A., Ferraris, S. P., Saha, S. P., Hessel, E. A. II, Haan, C. K., Royston, B. D., Bridges, C. R., Higgins, R. S.D., Despotis, G., Brown, J. R., The Society of Cardiovascular Anesthesiologists Sp, , Spiess, B. D., Shore-Lesserson, L., Stafford-Smith, M., Mazer, C. D., Bennett-Guerrero, E., Hill, S. E., Body, S.
(2007). Perioperative Blood Transfusion and Blood Conservation in Cardiac Surgery: The Society of Thoracic Surgeons and The Society of Cardiovascular Anesthesiologists Clinical Practice Guideline. Ann. Thorac. Surg.
83: S27-S86
[Abstract][Full Text]
Stukel, T. A., Fisher, E. S., Wennberg, D. E., Alter, D. A., Gottlieb, D. J., Vermeulen, M. J.
(2007). Analysis of Observational Studies in the Presence of Treatment Selection Bias: Effects of Invasive Cardiac Management on AMI Survival Using Propensity Score and Instrumental Variable Methods. JAMA
297: 278-285
[Abstract][Full Text]
Coulter, I. D.
(2006). Evidence Summaries and Synthesis: Necessary but Insufficient Approach for Determining Clinical Practice of Integrated Medicine?. Integr Cancer Ther
5: 282-286
[Abstract]
Butzkueven, H, Chapman, J, Cristiano, E, Grand'Maison, F, Hoffmann, M, Izquierdo, G, Jolley, D, Kappos, L, Leist, T, Pohlau, D, Rivera, V, Trojano, M, Verheul, F, Malkowski, J-P
(2006). MSBase: an international, online registry and platform for collaborative outcomes research in multiple sclerosis. Mult Scler
12: 769-774
[Abstract]
Bonovas, S., Filioussi, K., Tsavaris, N., Sitaras, N. M.
(2006). Statins and Cancer Risk: A Literature-Based Meta-Analysis and Meta-Regression Analysis of 35 Randomized Controlled Trials. JCO
24: 4808-4817
[Abstract][Full Text]
Grunkemeier, G. L., Jin, R., Starr, A.
(2006). Prosthetic heart valves: objective performance criteria versus randomized clinical trial.. Ann. Thorac. Surg.
82: 776-780
[Abstract][Full Text]
Sasson, L., Katz, M. G., Ezri, T., Tamir, A., Herman, A., Bove, E. L., Schachner, A.
(2006). Indications for tricuspid valve detachment in closure of ventricular septal defect in children.. Ann. Thorac. Surg.
82: 958-963
[Abstract][Full Text]
Trojano, M., Russo, P., Fuiani, A., Paolicelli, D., Di Monte, E., Granieri, E., Rosati, G., Savettieri, G., Comi, G., Livrea, P., MSDN Study Group,
(2006). The Italian Multiple Sclerosis Database Network (MSDN): the risk of worsening according to IFN{beta} exposure in multiple sclerosis. Mult Scler
12: 578-585
[Abstract]
Davis, W. A., Bruce, D. G., Davis, T. M.E.
(2006). Is Self-Monitoring of Blood Glucose Appropriate for All Type 2 Diabetic Patients?: The Fremantle Diabetes Study. Diabetes Care
29: 1764-1770
[Abstract][Full Text]
Tiihonen, J., Walhbeck, K., Lonnqvist, J., Klaukka, T., Ioannidis, J. P A, Volavka, J., Haukka, J.
(2006). Effectiveness of antipsychotic treatments in a nationwide cohort of patients in community care after first hospitalisation due to schizophrenia and schizoaffective disorder: observational follow-up study. BMJ
333: 224-
[Abstract][Full Text]
Cooper, N. J., Sutton, A. J., Lu, G., Khunti, K.
(2006). Mixed comparison of stroke prevention treatments in individuals with nonrheumatic atrial fibrillation.. Arch Intern Med
166: 1269-1275
[Abstract][Full Text]
Ganti, A. K., Mulshine, J. L.
(2006). Lung cancer screening.. The Oncologist
11: 481-487
[Abstract][Full Text]
El Ansari, W., Weiss, E. S
(2006). Quality of research on community partnerships: developing the evidence base. Health Educ Res
21: 175-180
[Abstract][Full Text]
Papanikolaou, P. N., Christidi, G. D., Ioannidis, J. P.A.
(2006). Comparison of evidence on harms of medical interventions in randomized and nonrandomized studies. CMAJ
174: 635-641
[Abstract][Full Text]
Concato, J., Wells, C. K., Horwitz, R. I., Penson, D., Fincke, G., Berlowitz, D. R., Froehlich, G., Blake, D., Vickers, M. A., Gehr, G. A., Raheb, N. H., Sullivan, G., Peduzzi, P.
(2006). The Effectiveness of Screening for Prostate Cancer: A Nested Case-Control Study. Arch Intern Med
166: 38-43
[Abstract][Full Text]
Reider, B.
(2005). The Search for Quality. Am J Sports Med
33: 1801-1802
[Full Text]
Koch, E., Otarola, A., Kirschbaum, A.
(2005). A landmark for popperian epidemiology: refutation of the randomised Aldactone evaluation study. J. Epidemiol. Community Health
59: 1000-1006
[Abstract][Full Text]
Greene, T., Daugirdas, J., Depner, T., Allon, M., Beck, G., Chumlea, C., Delmez, J., Gotch, F., Kusek, J. W., Levin, N., Owen, W., Schulman, G., Star, R., Toto, R., Eknoyan, G., for the Hemodialysis Study Group,
(2005). Association of Achieved Dialysis Dose with Mortality in the Hemodialysis Study: An Example of "Dose-Targeting Bias". J. Am. Soc. Nephrol.
16: 3371-3380
[Abstract][Full Text]
Cote, C. G., Celli, B. R.
(2005). Pulmonary rehabilitation and the BODE index in COPD. Eur Respir J
26: 630-636
[Abstract][Full Text]
Haas, J, Maas-Enriquez, M, Hartung, H P
(2005). Intravenous immunoglobulins in the treatment of relapsing remitting multiple sclerosis - results of a retrospective multicenter observational study over five years. Mult Scler
11: 562-567
[Abstract]
Bauer, T. T., Welte, T., Ernen, C., Schlosser, B. M., Thate-Waschke, I., de Zeeuw, J., Schultze-Werninghaus, G.
(2005). Cost Analyses of Community-Acquired Pneumonia From the Hospital Perspective. Chest
128: 2238-2246
[Abstract][Full Text]
Du, X. L., Jones, D. V., Zhang, D.
(2005). Effectiveness of Adjuvant Chemotherapy for Node-Positive Operable Breast Cancer in Older Women. Journals of Gerontology Series A: Biological Sciences and Medical Sciences
60: 1137-1144
[Abstract][Full Text]
Ioannidis, J. P. A.
(2005). Contradicted and Initially Stronger Effects in Highly Cited Clinical Research. JAMA
294: 218-228
[Abstract][Full Text]
Retsas, S.
(2005). In Reply:. JCO
23: 4469-4469
[Full Text]
Santaguida, P. L., Helfand, M., Raina, P.
(2005). Challenges in Systematic Reviews That Evaluate Drug Efficacy or Effectiveness. ANN INTERN MED
142: 1066-1072
[Abstract][Full Text]
Chou, R., Helfand, M.
(2005). Challenges in Systematic Reviews That Assess Treatment Harms. ANN INTERN MED
142: 1090-1099
[Abstract][Full Text]
Hartling, L., McAlister, F. A., Rowe, B. H., Ezekowitz, J., Friesen, C., Klassen, T. P.
(2005). Challenges in Systematic Reviews of Therapeutic Devices and Procedures. ANN INTERN MED
142: 1100-1111
[Abstract][Full Text]
Habib, R. H., Zacharias, A., Schwann, T. A., Riordan, C. J., Durham, S. J., Shah, A.
(2005). Effects of Obesity and Small Body Size on Operative and Long-Term Outcomes of Coronary Artery Bypass Surgery: A Propensity-Matched Analysis. Ann. Thorac. Surg.
79: 1976-1986
[Abstract][Full Text]
Hannan, E. L., Racz, M. J., Walford, G., Jones, R. H., Ryan, T. J., Bennett, E., Culliford, A. T., Isom, O. W., Gold, J. P., Rose, E. A.
(2005). Long-Term Outcomes of Coronary-Artery Bypass Grafting versus Stent Implantation. NEJM
352: 2174-2183
[Abstract][Full Text]
Patsopoulos, N. A., Analatos, A. A., Ioannidis, J. P. A.
(2005). Relative Citation Impact of Various Study Designs in the Health Sciences. JAMA
293: 2362-2366
[Abstract][Full Text]
Rochon, P. A, Gurwitz, J. H, Sykora, K., Mamdani, M., Streiner, D. L, Garfinkel, S., Normand, S.-L. T, Anderson, G. M
(2005). Reader's guide to critical appraisal of cohort studies: 1. Role and design. BMJ
330: 895-897
[Full Text]
Wei, L., Ebrahim, S., Bartlett, C., Davey, P. D, Sullivan, F. M, MacDonald, T. M
(2005). Statin use in the secondary prevention of coronary heart disease in primary care: cohort study and comparison of inclusion and outcome with patients in randomised trials. BMJ
330: 821-
[Abstract][Full Text]
King, M., Nazareth, I., Lampe, F., Bower, P., Chandler, M., Morou, M., Sibbald, B., Lai, R.
(2005). Impact of Participant and Physician Intervention Preferences on Randomized Trials: A Systematic Review. JAMA
293: 1089-1099
[Abstract][Full Text]
Seligman, S. J., Vagenakis, A. G., Kyriazopoulou, V., Marras, T. K., Thwaites, G. E., Quy, H. T., Farrar, J. J., Quagliarello, V.
(2005). Dexamethasone for Tuberculous Meningitis. NEJM
352: 628-630
[Full Text]
Dixon, J. B., Pories, W. J., O'Brien, P. E., Schauer, P. R., Zimmet, P.
(2005). Surgery as an Effective Early Intervention for Diabesity: Why the reluctance?. Diabetes Care
28: 472-474
[Full Text]
Greene, M. F.
(2004). Vaginal Birth after Cesarean Revisited. NEJM
351: 2647-2649
[Full Text]
Retsas, S.
(2004). Treatment at Random: The Ultimate Science or the Betrayal of Hippocrates?. JCO
22: 5005-5008
[Full Text]
Hanks, G W, Nugent, M, Higgs, C M B, Busch, M A
(2004). Oral transmucosal fentanyl citrate in the management of breakthrough pain in cancer: an open, multicentre, dose-titration and long-term use study. Palliat Med
18: 698-704
[Abstract]
Cole, P. A., Bhandari, M.
(2004). What's New in Orthopaedic Trauma. JBJS
86: 2782-2795
[Full Text]
Lee, S. S.-J., Liu, Y.-C., Chen, Y.-S., Tsai, H.-C., Wann, S.-R., Kao, C.-H., Chang, F.-Y., Siu, L. K.
(2004). Insufficient Evidence that Extended-Spectrum Cephalosporins Effectively Prevent Metastatic Infections Related to Klebsiella pneumoniae-Caused Liver Abscess. Antimicrob. Agents Chemother.
48: 3640-3641
[Full Text]
Previous
