Background In the hierarchy of research designs, the resultsof randomized, controlled trials are considered to be evidenceof the highest grade, whereas observational studies are viewedas having less validity because they reportedly overestimatetreatment effects. We used published meta-analyses to identifyrandomized clinical trials and observational studies that examinedthe same clinical topics. We then compared the results of theoriginal reports according to the type of research design.
Methods A search of the Medline data base for articles publishedin five major medical journals from 1991 to 1995 identifiedmeta-analyses of randomized, controlled trials and meta-analysesof either cohort or case–control studies that assessedthe same intervention. For each of five topics, summary estimatesand 95 percent confidence intervals were calculated on the basisof data from the individual randomized, controlled trials andthe individual observational studies.
Results For the five clinical topics and 99 reports evaluated,the average results of the observational studies were remarkablysimilar to those of the randomized, controlled trials. For example,analysis of 13 randomized, controlled trials of the effectivenessof bacille Calmette–Guérin vaccine in preventingactive tuberculosis yielded a relative risk of 0.49 (95 percentconfidence interval, 0.34 to 0.70) among vaccinated patients,as compared with an odds ratio of 0.50 (95 percent confidenceinterval, 0.39 to 0.65) from 10 case–control studies.In addition, the range of the point estimates for the effectof vaccination was wider for the randomized, controlled trials(0.20 to 1.56) than for the observational studies (0.17 to 0.84).
Conclusions The results of well-designed observational studies(with either a cohort or a case–control design) do notsystematically overestimate the magnitude of the effects oftreatment as compared with those in randomized, controlled trialson the same topic.
Randomized, controlled trials were introduced into clinicalmedicine when streptomycin was evaluated in the treatment oftuberculosis1 and have become the gold standard for assessingthe effectiveness of therapeutic agents.2,3,4 The ascendancyof randomized, controlled trials was hastened by a landmarkarticle5 comparing published randomized, controlled studieswith those that used observational designs. That review of theliterature identified six different therapies evaluated in bothrandomized, controlled trials (50 studies) and trials with historicalcontrols (56 studies). For each study, subjects in the treatmentgroup were found to have similar rates of the outcome in questionregardless of study design, but subjects in the control groupin trials with historical controls had worse outcomes than controlsubjects in randomized, controlled trials. The agent being testedwas considered effective in 44 of 56 trials with historicalcontrols (79 percent), but in only 10 of 50 randomized, controlledtrials (20 percent). The authors concluded that biases in patientselection may irretrievably weight the outcome of historicalcontrolled trials in favor of new therapies.5
Current criticisms of observational studies involve, in additionto trials with historical controls, cohort studies with concurrentselection of control subjects, as well as case–controldesigns. Advocates of "evidence-based medicine"6 classify studiesaccording to "grades of evidence" on the basis of the researchdesign, using internal validity (i.e., the correctness of theresults) as the criterion for hierarchical rankings. An exampleof such rankings is shown in Table 1. The highest grade is reservedfor research involving "at least one properly randomized controlledtrial," and the lowest grade is applied to descriptive studies(e.g., case series) and expert opinion; observational studies,both cohort studies and case–control studies, fall atintermediate levels.7 Although the quality of studies is sometimesevaluated within each grade, each category is considered methodologicallysuperior to those below it. This hierarchical approach to studydesign has been promoted widely in individual reports, meta-analyses,consensus statements, and educational materials for clinicians.
Table 1. Grades of Evidence for the Purported Quality of Study Design.
Systematic reviews and meta-analyses offer an opportunity totest implicit assumptions about the hierarchy of research designs.If particular associations between exposure and outcome werestudied in both randomized, controlled trials and cohort orcase–control studies, and if these studies were then includedin meta-analyses, the results could be compared according tostudy design, as was done for trials with historical controls.5In the current study, however, we evaluated only observationalstudies that used contemporaneous control subjects. The variationin point estimates of associations between exposure and outcomeprovides data to confirm or refute the assumptions regardingobservational studies, as well as the strengths and limitationsof a "design hierarchy."
Methods
We identified published reports of randomized, controlled trialsand reports of observational studies with either a cohort design(i.e., with concurrent selection of controls) or a case–controldesign that assessed the same clinical topic (clinical interventionand outcome). The articles were selected by first identifyingmeta-analyses published in five major journals (Annals of InternalMedicine, the British Medical Journal, the Journal of AmericanMedical Association, the Lancet, and the New England Journalof Medicine) from 1991 to 1995. The meta-analyses were identifiedby searching Medline for the terms "meta-analysis," "meta-analyses,""pooling," "combining," "overview," and "aggregation." Additionalreferences were found in Current Contents, supplemented by searchesof printed copies of the relevant journals.
The meta-analyses were then classified, by consensus of twoinvestigators, as including randomized, controlled trials only,observational studies only, or both. Clinical trials were definedas studies that used random assignment of interventions; observationalstudies had either cohort or case–control designs. Meta-analyseswere excluded if they involved cohort studies with historicalcontrols or clinical trials with nonrandom assignment of interventions,or if they did not report results in the format of point estimates(e.g., relative risks or odds ratios) and confidence intervals.In this context, odds ratios and relative risks will be similarin magnitude, because the rates of the outcome events are low.The remaining meta-analyses were then reviewed, and the originalstudies cited in the bibliographies were retrieved. Althoughthe meta-analyses themselves often used criteria related toquality in the selection of studies, we also evaluated the originalreports, using published scoring criteria.8,9,10
We performed two main analyses, the results of which are reportedhere. In the first analysis, summary estimates and 95 percentconfidence intervals were determined for each clinical topic,according to whether the data came from randomized, controlledtrials or observational studies. Pooled analyses were performedaccording to the method of DerSimonian and Laird11 as describedby Fleiss.12 We chose the random-effects model for combiningdata because it provides more conservative results (wider confidenceintervals) than a fixed-effects model. When possible, pooledestimates were computed on the basis of data from the originalreports. If such estimates were not available, however, thedata from the published reports of the meta-analyses were used.For example, the meta-analysis of cohort studies involving blood-pressuremeasurements13 used an adjustment for bias due to regressiondilution (regression toward the mean) for the calculation ofpoint estimates and 95 percent confidence intervals. A secondanalysis was used to describe the range of results from theindividual studies; such a range is presented for each clinicaltopic, with the studies grouped according to research design.
Results
The Studies
The search strategy yielded 102 citations of meta-analyses (availablefrom the authors on request), including 6 that examined bothrandomized, controlled trials and observational studies of thesame clinical topic. The remaining 96 meta-analyses includedrandomized, controlled trials only (72 reports) or observationalstudies only (24 reports). Among these reports, three additionalclinical topics were identified for which each type of studydesign was assessed separately. The nine clinical topics (atotal of 12 meta-analyses) included five that met our eligibilitycriteria and provided the data for the current analysis. Theremaining four topics (data not shown) were excluded becausethey were the subject of observational studies with only historicalcontrols or because no data were available in the form of pointestimates.
The five clinical topics (Table 2) were investigated in 99 originalarticles with a total of 1,871,681 study subjects. The pooled(summary) point estimates are presented in Table 2, and therange of point estimates from each study, when available, isshown in Figure 1. Among the 99 studies, 6 randomized, controlledtrials (6 percent) contributed data to the summary results butdid not have a quantitative point estimate for the main associationof interest (because no outcomes were observed in one or bothof the compared groups).
Figure 1. Range of Point Estimates According to Type of Research Design.
The studies evaluated bacille Calmette–Guérin vaccine and active tuberculosis (13 randomized, controlled trials and 10 case–control studies), screening mammography and mortality from breast cancer (8 randomized, controlled trials and 4 case–control studies), cholesterol levels and death due to trauma among men (4 of 6 randomized, controlled trials [2 trials did not provide point estimates]; the results of the 14 cohort studies were not reported individually), treatment of hypertension and stroke among only the men in the studies (11 randomized, controlled trials and 7 cohort studies), and treatment of hypertension and coronary heart disease among only the men in the studies (13 of 14 randomized, controlled trials [1 trial did not provide point estimates] and 9 cohort studies). Solid circles indicate randomized, controlled trials, and open circles observational studies.
Bacille Calmette–Guérin Vaccine and Tuberculosis
The effectiveness of bacille Calmette–Guérin vaccineagainst active tuberculosis was examined in a meta-analysis14of 13 randomized trials (with 359,922 subjects), yielding apooled relative risk of 0.49 (95 percent confidence interval,0.34 to 0.70), and 10 case–control studies (with 6511subjects), yielding a pooled odds ratio of 0.50 (95 percentconfidence interval, 0.39 to 0.65). The point estimates fromthe original articles ranged from 0.20 to 1.56 for randomized,controlled trials and from 0.17 to 0.84 for observational studies.
Screening Mammography and Mortality from Breast Cancer
A meta-analysis15 of eight randomized trials (with 429,043 subjects)of the relation between screening mammography and mortalityfrom breast cancer found a protective effect of screening amongwomen 40 years of age or older, with a pooled relative riskof 0.79 (95 percent confidence interval, 0.71 to 0.88); a benefitof screening was also found in four case–control studies(with 132,456 subjects), with a pooled odds ratio of 0.61 (95percent confidence interval, 0.49 to 0.77). The range of pointestimates was 0.68 to 0.97 among randomized, controlled trialsand 0.51 to 0.76 among observational studies.
Cholesterol Levels and Death Due to Trauma
A meta-analysis16 of six randomized, controlled trials (with36,910 men) reported a pooled relative risk of death due totrauma of 1.42 (95 percent confidence interval, 0.94 to 2.15),indicating an increased risk among subjects taking the drugsthat were studied. A separate meta-analysis17 of 14 cohort studies(9377 subjects) reported a pooled hazard ratio of 1.40 (95 percentconfidence interval, 1.14 to 1.66). The range of point estimatesfrom the cohort studies was not reported, precluding a comparisonwith the range of results from the randomized, controlled trials(the range was 0.25 to 2.74 in four randomized, controlled trials;two trials did not report quantitative results).
Treatment of Hypertension and Stroke
The relation between the treatment of hypertension and a firstoccurrence of stroke (i.e., the effectiveness of primary prevention)was examined in meta-analyses of 14 randomized, controlled trials18and 7 cohort studies.13 The pooled results from the randomized,controlled trials (36,894 subjects) yielded a point estimateof the risk of stroke of 0.58 (95 percent confidence interval,0.50 to 0.67) among patients given antihypertensive treatment;the pooled results from the observational studies (405,511 subjects)yielded an adjusted point estimate of 0.62 (95 percent confidenceinterval, 0.60 to 0.65). The range of results was 0.24 to 1.91for randomized, controlled trials (three of the randomized,controlled trials did not provide point estimates) and 0.49to 0.58 (unadjusted values) for cohort studies.
Treatment of Hypertension and Coronary Heart Disease
A meta-analysis18 of 14 randomized, controlled trials (36,894subjects) reported a pooled point estimate of the relative riskof coronary heart disease of 0.86 (95 percent confidence interval,0.78 to 0.96) among patients treated for hypertension, and ameta-analysis13 of 9 cohort studies (418,343 subjects) reportedan adjusted, pooled point estimate of 0.77 (95 percent confidenceinterval, 0.75 to 0.80). The range of results was 0.49 to 1.60for randomized, controlled trials (one randomized, controlledtrial did not report a relative risk) and 0.65 to 0.72 (unadjustedvalues) for cohort studies.
Discussion
Our results challenge the current consensus about a hierarchyof study designs in clinical research. Contrary to prevailingbeliefs, the "average results" from well-designed observationalstudies (with a cohort or case–control design) did notsystematically overestimate the magnitude of the associationsbetween exposure and outcome as compared with the results ofrandomized, controlled trials of the same topic. Rather, thesummary results of randomized, controlled trials and observationalstudies were remarkably similar for each clinical topic we examined(Table 2). Viewed individually, the observational studies hadless variability in point estimates (i.e., less heterogeneityof results) than randomized, controlled trials on the same topic(Figure 1). In fact, only among randomized, controlled trialsdid some studies report results in a direction opposite thatof the pooled point estimate, representing a paradoxical finding(e.g., treatment of hypertension was unexpectedly associatedwith higher rates of coronary heart disease in several clinicaltrials).
Although the data we present are a challenge to accepted beliefs,the findings are consistent with three other types of availableevidence. For example, previous investigations have shown thatobservational cohort studies can produce results similar tothose of randomized, controlled trials when similar criteriaare used to select study subjects. In addition, data from nonmedicalresearch do not support a hierarchy of research designs. Finally,the finding that there is substantial variation in the resultsof randomized, controlled trials is consistent with prior evidenceof contradictory results among randomized, controlled trials.
First, there is evidence that observational studies can be designedwith rigorous methods that mimic those of clinical trials andthat well-designed observational studies do not consistentlyoverestimate the effectiveness of therapeutic agents. An analysis19of 18 randomized and observational studies in health-servicesresearch found that treatment effects may differ according toresearch design, but that "one method does not give a consistentlygreater effect than the other." The treatment effects were mostsimilar when the exclusion criteria were similar and when theprognostic factors were accounted for in observational studies.
A specific method used to strengthen observational studies (the"restricted cohort" design9) adapts principles of the designof randomized, controlled trials to the design of an observationalstudy as follows: it identifies a "zero time" for determininga patient's eligibility and base-line features, uses inclusionand exclusion criteria similar to those of clinical trials,adjusts for differences in base-line susceptibility to the outcome,and uses statistical methods (e.g., intention-to-treat analysis)similar to those of randomized, controlled trials. When theseprocedures were used in a cohort study9 evaluating the benefitof beta-blockers after recovery from myocardial infarction,the use of a restricted cohort produced results consistent withcorresponding findings from the Beta-Blocker Heart Attack Trial20:the three-year reductions in mortality were 33 percent and 28percent, respectively.
Second, data in the literature of other scientific disciplinessupport our contention that research design should not be considereda rigid hierarchy. A comprehensive review of research on variouspsychological, educational, and behavioral treatments21 identified302 meta-analyses and examined the reports on the basis of severalfeatures, including research design. Results were presentedfrom the 74 meta-analyses that included studies with randomizedand observational designs. To allow for comparisons among varioustopics with different outcome variables, effect size was usedas a unit-free measure of the effect of the intervention. Theobservational designs did not consistently overestimate or underestimatethe effect of treatment; the mean value of the difference wasa trivial 0.05. Thus, these independent data do not supportthe contention that observational studies overestimate effectsas compared with randomized, controlled trials.
Third, a review of more than 200 randomized, controlled trialson 36 clinical topics found numerous examples of conflictingresults.22 A more recent example is offered by studies addressingwhether therapy with monoclonal antibodies improves outcomesamong patients with septic shock (reviewed by Horn23 and Anguset al.24). In addition, one study25 found that the results ofmeta-analyses based on randomized, controlled trials were oftendiscordant with those of large, simple trials on the same clinicaltopic. Regardless of the reasons why randomized, controlledtrials produce heterogeneous results, the available evidenceindicates that a single randomized trial (or only one observationalstudy) cannot be expected to provide a gold-standard resultthat applies to all clinical situations.
One possible explanation for the finding that observationalstudies may be less prone to heterogeneity in results than randomized,controlled trials is that each observational study is more likelyto include a broad representation of the population at risk.In addition, there is less opportunity for differences in themanagement of subjects among observational studies. For example,although there is general agreement that physicians do not usetherapeutic agents in a uniform way, an observational studywould usually include patients with coexisting illnesses anda wide spectrum of disease severity, and treatment would betailored to the individual patient. In contrast, each randomized,controlled trial may have a distinct group of patients as aresult of specific inclusion and exclusion criteria regardingcoexisting illnesses and severity of disease, and the experimentalprotocol for therapy may not be representative of clinical practice.
The relevance of our findings extends beyond their implicationsfor expert panels such as the U.S. Preventive Services TaskForce.7 A popular "users' guide" for clinicians26 warns that"the potential for bias is much greater in cohort and case–controlstudies than in randomized, controlled trials, [so that] recommendationsfrom overviews combining observational studies will be muchweaker." The studies cited to support that claim27,28 (and similarclaims29), however, compare randomized, controlled trials withtrials using historical controls, unblinded clinical trials,or clinical trials without randomly assigned control subjects— not with the types of cohort and case–controlstudies included in our investigation. Thus, data based on "weaker"forms of observational studies are often mistakenly used tocriticize all observational research.
We examined the possibility that the quality of individual studiescould explain our findings. For example, randomized, controlledtrials that did not satisfy criteria with respect to qualitycould be the source of variability in point estimates, or theobservational studies might be of uniformly high quality. Whenstandard assessments of quality were applied to the studies,however, no association was found between the number of criteriafor high-quality research that a study satisfied and the rankorder of its point estimate (data not shown). Thus, althoughquality scores have been used in some situations to separatehigh-quality from low-quality randomized, controlled trials,8our results are consistent with other studies30 that did notfind an association between summary measures of quality andtreatment effects. The issue of how to judge the validity ofeach study (in terms of the methodologic aspects relevant toeach investigation) is beyond the scope of this report. However,judging validity is often not as simple as identifying the typeof research design or assessing general characteristics of thestudy.8,26
The meta-analyses of randomized, controlled trials and observationalstudies that we evaluated included single reports that combinedthe two types of research design, as well as separate reportsfor each category (Table 2). This mix of reports offers reassurancethat our findings are not attributable to the methods used ineach meta-analysis. (The overall paucity of meta-analyses includingboth randomized, controlled trials and observational studiesof the same research topic is consistent with our premise thatobservational studies are not considered trustworthy and thatthey are therefore not included in such investigations.) Thevalidity of our analysis is also supported by another investigationcomparing randomized, controlled trials and observational studiesof screening mammography that found results similar to ours.31
Despite the consistency of our results (involving five clinicaltopics and 99 separate studies), as well as the confirmatoryevidence available in the literature, we believe that the appropriaterole of observational studies may vary in different situations.For example, observational investigations of some kinds of treatments(e.g., surgical operations and other invasive therapies) maybe more prone to selection bias than the observational studiesof drugs and noninvasive tests that we examined in this study,and "softer" outcomes (e.g., functional status) may be morereadily assessed in randomized, controlled trials. In addition,we are aware of the risk that the results of poorly done observationalstudies may be used inappropriately — for example,32 topromote ineffective alternative therapies.
Randomized, controlled trials will (and should) remain a prominenttool in clinical research, but the results of a single randomized,controlled trial, or of only one observational study, shouldbe interpreted cautiously. If a randomized, controlled trialis later determined to have given wrong answers, evidence bothfrom other trials and from well-designed cohort or case–controlstudies can and should be used to find the right answers. Thepopular belief that only randomized, controlled trials producetrustworthy results and that all observational studies are misleadingdoes a disservice to patient care, clinical investigation, andthe education of health care professionals.
Dr. Concato is the recipient of a Career Development Award fromthe Veterans Affairs Health Services Research and DevelopmentService.
Source Information
From the Departments of Internal Medicine (J.C., N.S., R.I.H.) and Epidemiology and Public Health (R.I.H.), Yale University School of Medicine, New Haven, Conn.; and the Clinical Epidemiology Unit, West Haven Veterans Affairs Medical Center, West Haven, Conn. (J.C.).
Address reprint requests to Dr. Concato at the Department of Internal Medicine, Yale University School of Medicine, 333 Cedar St., SHM IE-61, New Haven, CT 06510, or at john.concato{at}yale.edu.
References
Streptomycin treatment of pulmonary tuberculosis: a Medical Research Council investigation. BMJ 1948;2:769-782. [Free Full Text]
Byar DP, Simon RM, Friedewald WT, et al. Randomized clinical trials: perspectives on some recent ideas. N Engl J Med 1976;295:74-80. [Abstract]
Feinstein AR. Current problems and future challenges in randomized clinical trials. Circulation 1984;70:767-774. [Free Full Text]
Abel U, Koch A. The role of randomization in clinical studies: myths and beliefs. J Clin Epidemiol 1999;52:487-497. [CrossRef][Medline]
Sacks H, Chalmers TC, Smith H Jr. Randomized versus historical controls for clinical trials. Am J Med 1982;72:233-240. [CrossRef][Medline]
Evidence-Based Medicine Working Group. Evidence-based medicine: a new approach to teaching the practice of medicine. JAMA 1992;268:2420-2425. [Free Full Text]
Preventive Services Task Force. Guide to clinical preventive services: report of the U.S. Preventive Services Task Force. 2nd ed. Baltimore: Williams & Wilkins, 1996.
Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408-412. [Free Full Text]
Horwitz RI, Viscoli CM, Clemens JD, Sadock RT. Developing improved observational methods for evaluating therapeutic effectiveness. Am J Med 1990;89:630-638. [CrossRef][Medline]
Horwitz RI, Feinstein AR. Methodologic standards and contradictory results in case-control research. Am J Med 1979;66:556-564. [CrossRef][Medline]
DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials 1986;7:177-188. [CrossRef][Medline]
Fleiss JL. The statistical basis of meta-analysis. Stat Methods Med Res 1993;2:121-145. [Medline]
MacMahon S, Peto R, Cutler J, et al. Blood pressure, stroke, and coronary heart disease. 1. Prolonged differences in blood pressure: prospective observational studies corrected for the regression dilution bias. Lancet 1990;335:765-774. [CrossRef][Medline]
Colditz GA, Brewer TF, Berkey CS, et al. Efficacy of BCG vaccine in the prevention of tuberculosis: meta-analysis of the published literature. JAMA 1994;271:698-702. [Free Full Text]
Kerlikowske K, Grady D, Rubin SM, Sandrock C, Ernster VL. Efficacy of screening mammography: a meta-analysis. JAMA 1995;273:149-154. [Free Full Text]
Cummings P, Psaty BM. The association between cholesterol and death from injury. Ann Intern Med 1994;120:848-855. [Free Full Text]
Jacobs D, Blackburn H, Higgins M, et al. Report of the Conference on Low Blood Cholesterol: mortality associations. Circulation 1992;86:1046-1060. [Free Full Text]
Collins R, Peto R, MacMahon S, et al. Blood pressure, stroke, and coronary heart disease. 2. Short-term reductions in blood pressure: overview of randomised drug trials in their epidemiological context. Lancet 1990;335:827-838. [CrossRef][Medline]
McKee M, Britton A, Black N, McPherson K, Sanderson C, Bain C. Methods in health services research: interpreting the evidence: choosing between randomised and non-randomised studies. BMJ 1999;319:312-315. [Free Full Text]
A randomized trial of propranolol in patients with acute myocardial infarction. I. Mortality results. JAMA 1982;247:1707-1714. [Free Full Text]
Lipsey MW, Wilson DB. The efficacy of psychological, educational, and behavioral treatment: confirmation from meta-analysis. Am Psychol 1993;48:1181-1209. [CrossRef][Medline]
Horwitz RI. Complexity and contradiction in clinical trial research. Am J Med 1987;82:498-510. [CrossRef][Medline]
Horn KD. Evolving strategies in the treatment of sepsis and systemic inflammatory response syndrome (SIRS). QJM 1998;91:265-277. [Free Full Text]
Angus DC, Birmingham MC, Balk RA, et al. E5 murine monoclonal antiendotoxin antibody in gram-negative sepsis: a randomized controlled trial. JAMA 2000;283:1723-1730. [Free Full Text]
LeLorier J, Grégoire G, Benhaddad A, Lapierre J, Derderian F. Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med 1997;337:536-542. [Free Full Text]
Guyatt GH, Sackett DL, Sinclair JC, Hayward R, Cook DJ, Cook RJ. Users' guide to the medical literature. IX. A method for grading health care recommendations. JAMA 1995;274:1800-1804. [Erratum, JAMA 1996;275:1232.] [Free Full Text]
Chalmers TC, Celano P, Sacks HS, Smith H Jr. Bias in treatment assignment in controlled clinical trials. N Engl J Med 1983;309:1358-1361. [Abstract]
Sacks HS, Chalmers TC, Smith H Jr. Sensitivity and specificity of clinical trials: randomized v historical controls. Arch Intern Med 1983;143:753-755. [Free Full Text]
Kunz R, Oxman AD. The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials. BMJ 1998;317:1185-1190. [Free Full Text]
Juni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 1999;282:1054-1060. [Free Full Text]
Demissie K, Mills OF, Rhoads GG. Empirical comparison of the results of randomized controlled trials and case-control studies in evaluating the effectiveness of screening mammography. J Clin Epidemiol 1998;51:81-91. [CrossRef][Medline]
Angell M, Kassirer JP. Alternative medicine -- the risks of untested and unregulated remedies. N Engl J Med 1998;339:839-841. [Free Full Text]
Observational Studies and Randomized Trials
Kunz R., Khan K. S., Neumayer H.-H., Sacks H. S., Liu P.-Y., Anderson G., Crowley J. J., Friedman H. S., Smith R. P., Meier P., Benson K., Hartz A. J., Concato J., Shah N., Horwitz R. I.
Extract |
Full Text
N Engl J Med 2000;
343:1194-1197, Oct 19, 2000.
Correspondence
Ferreira-Gonzalez, I., Marsal, J. R., Mitjavila, F., Parada, A., Ribera, A., Cascant, P., Soriano, N., Sanchez, P. L., Aros, F., Heras, M., Bueno, H., Marrugat, J., Cunat, J., Civeira, E., Permanyer-Miralda, G.
(2009). Patient Registries of Acute Coronary Syndrome: Assessing or Biasing the Clinical Real World Data?. Circ Cardiovasc Qual Outcomes
2: 540-547
[Abstract][Full Text]
Mazzaglia, G., Ambrosioni, E., Alacqua, M., Filippi, A., Sessa, E., Immordino, V., Borghi, C., Brignoli, O., Caputi, A. P., Cricelli, C., Mantovani, L. G.
(2009). Adherence to Antihypertensive Medications and Cardiovascular Morbidity Among Newly Diagnosed Hypertensive Patients. Circulation
120: 1598-1605
[Abstract][Full Text]
La Caze, A.
(2009). Evidence-Based Medicine Must Be .... J Med Philos
34: 509-527
[Abstract][Full Text]
Chan, K. E., Lazarus, J. M., Thadhani, R., Hakim, R. M.
(2009). Warfarin Use Associates with Increased Risk for Stroke in Hemodialysis Patients with Atrial Fibrillation. J. Am. Soc. Nephrol.
20: 2223-2233
[Abstract][Full Text]
Kolve, H., Ahlke, E., Fegeler, W., Ritter, J., Jurgens, H., Groll, A. H.
(2009). Safety, tolerance and outcome of treatment with liposomal amphotericin B in paediatric patients with cancer or undergoing haematopoietic stem cell transplantation. J Antimicrob Chemother
64: 383-387
[Abstract][Full Text]
Federman, D. G., Concato, J., Kirsner, R. S.
(2009). Screening for Skin Cancer: Absence of Evidence. Arch Dermatol
145: 926-927
[Full Text]
Concato, J., Horwitz, R. I.
(2009). Clinical Practice Guidelines and Scientific Evidence. JAMA
302: 145-145
[Full Text]
Tabet, N., Howard, R.
(2009). Non-pharmacological interventions in the prevention of delirium. Age Ageing
38: 374-379
[Abstract][Full Text]
Huynh, T., Perron, S., O'Loughlin, J., Joseph, L., Labrecque, M., Tu, J. V., Theroux, P.
(2009). Comparison of Primary Percutaneous Coronary Intervention and Fibrinolytic Therapy in ST-Segment-Elevation Myocardial Infarction: Bayesian Hierarchical Meta-Analyses of Randomized Controlled Trials and Observational Studies. Circulation
119: 3101-3109
[Abstract][Full Text]
Bilimoria, K. Y., Bentrem, D. J., Hansen, N. M., Bethke, K. P., Rademaker, A. W., Ko, C. Y., Winchester, D. P., Winchester, D. J.
(2009). Comparison of Sentinel Lymph Node Biopsy Alone and Completion Axillary Lymph Node Dissection for Node-Positive Breast Cancer. JCO
27: 2946-2953
[Abstract][Full Text]
Weinstein, J. N., Lurie, J. D., Tosteson, T. D., Zhao, W., Blood, E. A., Tosteson, A. N.A., Birkmeyer, N., Herkowitz, H., Longley, M., Lenke, L., Emery, S., Hu, S. S.
(2009). Surgical Compared with Nonoperative Treatment for Lumbar Degenerative Spondylolisthesis. Four-Year Results in the Spine Patient Outcomes Research Trial (SPORT) Randomized and Observational Cohorts. JBJS
91: 1295-1304
[Abstract][Full Text]
Doi, S. A. R., Onitilo, A.
(2009). Author Reply. Clin Med Res
7: 2-3
[Full Text]
Barrios, V., Escobar, C.
(2009). Importance of Blood Pressure Control in Hypertensive Patients With Coronary Heart Disease in Clinical Practice to Reduce the Risk of Stroke. Stroke
40: e469-e469
[Full Text]
Pepine, C. J., Cooper-DeHoff, R., Coca, A., Messerli, F., Benetos, A.
(2009). Response to Letter by Barrios and Escobar. Stroke
40: e470-e470
[Full Text]
Van der Niepen, P., Woestenburg, A., Brie, H., Vancayzeele, S., MacDonald, K., Denhaerynck, K., Lee, C., Hermans, C., Abraham, I.
(2009). Effectiveness of Valsartan for Treatment of Hypertension: Patient Profiling and Hierarchical Modeling of Determinants and Outcomes (the PREVIEW Study). The Annals of Pharmacotherapy
43: 849-861
[Abstract][Full Text]
Bhandari, M., Morshed, S., Tornetta, P. III, Schemitsch, E. H.
(2009). Design, Conduct, and Interpretation of Nonrandomized Orthopaedic Studies: A Practical Approach. (All) Evidence Matters. JBJS
91: 1-1
[Full Text]
Hoppe, D. J., Schemitsch, E. H., Morshed, S., Tornetta, P. III, Bhandari, M.
(2009). Hierarchy of Evidence: Where Observational Studies Fit in and Why We Need Them. JBJS
91: 2-9
[Abstract][Full Text]
Simunovic, N., Sprague, S., Bhandari, M.
(2009). Methodological Issues in Systematic Reviews and Meta-Analyses of Observational Studies in Orthopaedic Research. JBJS
91: 87-94
[Abstract][Full Text]
Li, Y., Zheng, Z., Xu, B., Zhang, S., Li, W., Gao, R., Hu, S.
(2009). Comparison of Drug-Eluting Stents and Coronary Artery Bypass Surgery for the Treatment of Multivessel Coronary Disease: Three-Year Follow-Up Results From a Single Institution. Circulation
119: 2040-2050
[Abstract][Full Text]
Chan, K. E., Lazarus, J. M., Thadhani, R., Hakim, R. M.
(2009). Anticoagulant and Antiplatelet Usage Associates with Mortality among Hemodialysis Patients. J. Am. Soc. Nephrol.
20: 872-881
[Abstract][Full Text]
Ziegler, R.
(2009). Mistletoe Preparation Iscador: Are there Methodological Concerns with Respect to Controlled Clinical Trials?. Evid Based Complement Alternat Med
6: 19-30
[Abstract][Full Text]
Ross, J. S., Gross, C. P.
(2009). Policy Research: Using Evidence to Improve Healthcare Delivery Systems. Circulation
119: 891-898
[Full Text]
Tannen, R. L, Weiner, M. G, Xie, D.
(2009). Use of primary care electronic medical record database in drug efficacy research on cardiovascular outcomes: comparison of database and randomised controlled trial findings. BMJ
338: b81-b81
[Abstract][Full Text]
Srivastava, R., Downey, E. C., O'Gorman, M., Feola, P., Samore, M., Holubkov, R., Mundorff, M., James, B. C., Rosenbaum, P., Young, P. C., Dean, J. M.
(2009). Impact of Fundoplication Versus Gastrojejunal Feeding Tubes on Mortality and in Preventing Aspiration Pneumonia in Young Children With Neurologic Impairment Who Have Gastroesophageal Reflux Disease. Pediatrics
123: 338-345
[Abstract][Full Text]
Yu, P. P.
(2009). The Role of Electronic Health Records in the Treatment of Complex Diseases at Multidisciplinary Health Care Systems. Am Soc Clin Oncol Ed Book
2009: 636-639
[Abstract][Full Text]
Simone, C. B., Simone, C. B. II
(2008). Re: Should Supplemental Antioxidant Administration Be Avoided During Chemotherapy and Radiation Therapy?. JNCI J Natl Cancer Inst
100: 1558-1559
[Full Text]
Tessitore, N., Bedogna, V., Poli, A., Mantovani, W., Lipari, G., Baggio, E., Mansueto, G., Lupo, A.
(2008). Adding access blood flow surveillance to clinical monitoring reduces thrombosis rates and costs, and improves fistula patency in the short term: a controlled cohort study. Nephrol Dial Transplant
23: 3578-3584
[Abstract][Full Text]
Anderson, P. A., McCormick, P. C., Angevine, P. D.
(2008). Randomized Controlled Trials of the Treatment of Lumbar Disk Herniation: 1983-2007. J Am Acad Orthop Surg
16: 566-573
[Abstract][Full Text]
Siempos, I. I., Vardakas, K. Z., Kopterides, P., Falagas, M. E.
(2008). Adjunctive therapies for community-acquired pneumonia: a systematic review. J Antimicrob Chemother
62: 661-668
[Abstract][Full Text]
Leichsenring, F., Rabung, S.
(2008). Effectiveness of Long-term Psychodynamic Psychotherapy: A Meta-analysis. JAMA
300: 1551-1565
[Abstract][Full Text]
Makela, K. T., Eskelinen, A., Pulkkinen, P., Paavolainen, P., Remes, V.
(2008). Total Hip Arthroplasty for Primary Osteoarthritis in Patients Fifty-five Years of Age or Older. An Analysis of the Finnish Arthroplasty Registry. JBJS
90: 2160-2170
[Abstract][Full Text]
Martens, E. P, Pestman, W. R, de Boer, A., Belitser, S. V, Klungel, O. H
(2008). Systematic differences in treatment effect estimates between propensity score methods and logistic regression. Int J Epidemiol
37: 1142-1147
[Abstract][Full Text]
Nallamothu, B. K., Hayward, R. A., Bates, E. R.
(2008). Beyond the Randomized Clinical Trial: The Role of Effectiveness Studies in Evaluating Cardiovascular Therapies. Circulation
118: 1294-1303
[Full Text]
Marsh, J.L., McMaster, W., Parvizi, J., Katz, S. I., Spindler, K.
(2008). AOA Symposium. Barriers (Threats) to Clinical Research. JBJS
90: 1769-1776
[Full Text]
Hannan, E. L.
(2008). Randomized Clinical Trials and Observational Studies: Guidelines for Assessing Respective Strengths and Limitations. J Am Coll Cardiol Intv
1: 211-217
[Abstract][Full Text]
Campo, A. E., Williams, V., Williams, R. B., Segundo, M. A., Lydston, D., Weiss, S. M.
(2008). Effects of LifeSkills Training on Medical Students' Performance in Dealing with Complex Clinical Cases. Acad. Psychiatry
32: 188-193
[Abstract][Full Text]
Beck, E J, Mandalia, S, Youle, M, Brettle, R, Fisher, M, Gompels, M, Kinghorn, G, McCarron, B, Pozniak, A, Tang, A, Walsh, J, Williams, I, Gazzard, B
(2008). Treatment outcome and cost-effectiveness of different highly active antiretroviral therapy regimens in the UK (1996-2002). Int J STD AIDS
19: 297-304
[Abstract][Full Text]
Park, D.-W., Yun, S.-C., Lee, S.-W., Kim, Y.-H., Lee, C. W., Hong, M.-K., Kim, J.-J., Choo, S. J., Song, H., Chung, C. H., Lee, J.-W., Park, S.-W., Park, S.-J.
(2008). Long-Term Mortality After Percutaneous Coronary Intervention With Drug-Eluting Stent Implantation Versus Coronary Artery Bypass Surgery for the Treatment of Multivessel Coronary Artery Disease. Circulation
117: 2079-2086
[Abstract][Full Text]
Weinstein, J. N., Tosteson, T. D., Lurie, J. D., Tosteson, A. N.A., Blood, E., Hanscom, B., Herkowitz, H., Cammisa, F., Albert, T., Boden, S. D., Hilibrand, A., Goldberg, H., Berven, S., An, H., the SPORT Investigators,
(2008). Surgical versus Nonsurgical Therapy for Lumbar Spinal Stenosis. NEJM
358: 794-810
[Abstract][Full Text]
Sanders, M. H., Montserrat, J. M., Farre, R., Givelber, R. J.
(2008). Positive Pressure Therapy: A Perspective on Evidence-based Outcomes and Methods of Application. Proc Am Thorac Soc
5: 161-172
[Abstract][Full Text]
Hemila, H.
(2008). THE AUTHOR REPLIES. Am J Epidemiol
167: 501-502
[Full Text]
Simpson, R. J Jr
(2008). Assessing the safety of drugs through observational research. Heart
94: 129-130
[Full Text]
Morshed, S., Bhandari, M.
(2008). Clinical Trial Design in Fracture-Healing Research: Meeting the Challenge. JBJS
90: 55-61
[Abstract][Full Text]
Ferraris, V. A., Edwards, F. H., Shahian, D. M., Ferraris, S. P.
(2008). Risk Stratification and Comorbidity. Card Surg Adult
3: 199-246
[Full Text]
August, D. A., Serrano, D.
(2007). Outcomes Research in Specialized Nutrition Support. Nutr Clin Pract
22: 602-608
[Abstract][Full Text]
Shrier, I., Boivin, J.-F., Steele, R. J., Platt, R. W., Furlan, A., Kakuma, R., Brophy, J., Rossignol, M.
(2007). Should Meta-Analyses of Interventions Include Observational Studies in Addition to Randomized Controlled Trials? A Critical Examination of Underlying Principles. Am J Epidemiol
166: 1203-1209
[Abstract][Full Text]
Nicholl, J.
(2007). Case-mix adjustment in non-randomised observational evaluations: the constant risk fallacy. J. Epidemiol. Community Health
61: 1010-1013
[Abstract][Full Text]
Ravani, P., Parfrey, P. S., Dicks, E., Barrett, B. J.
(2007). Clinical research of kidney diseases II: problems of study design. Nephrol Dial Transplant
22: 2785-2794
[Full Text]
Worrall, J.
(2007). Why There's No Cause to Randomize. Br J Philos Sci
58: 451-488
[Abstract][Full Text]
Minkovitz, C. S., Strobino, D., Mistry, K. B., Scharfstein, D. O., Grason, H., Hou, W., Ialongo, N., Guyer, B.
(2007). Healthy Steps for Young Children: Sustained Results at 5.5 Years. Pediatrics
120: e658-e668
[Abstract][Full Text]
Blumenfeld, Z.
(2007). How to Preserve Fertility in Young Women Exposed to Chemotherapy? The Role of GnRH Agonist Cotreatment in Addition to Cryopreservation of Embrya, Oocytes, or Ovaries. The Oncologist
12: 1044-1054
[Abstract][Full Text]
Olivotto, I., Ommen, S. R., Maron, M. S., Cecchi, F., Maron, B. J.
(2007). Surgical Myectomy Versus Alcohol Septal Ablation for Obstructive Hypertrophic Cardiomyopathy: Will There Ever Be a Randomized Trial?. J Am Coll Cardiol
50: 831-834
[Abstract][Full Text]
Shojania, K. G., Sampson, M., Ansari, M. T., Ji, J., Doucette, S., Moher, D.
(2007). How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis. ANN INTERN MED
147: 224-233
[Abstract][Full Text]
Auerbach, A. D., Landefeld, C. S., Shojania, K. G.
(2007). The Tension between Needing to Improve Care and Knowing How to Do It. NEJM
357: 608-613
[Full Text]
Berry, D. A., Ravdin, P. M.
(2007). Breast Cancer Trends: A Marriage Between Clinical Trial Evidence and Epidemiology. JNCI J Natl Cancer Inst
99: 1139-1141
[Full Text]
Burns, T.
(2007). Evolution of outcome measures in schizophrenia. Br. J. Psychiatry
191: s1-s6
[Abstract][Full Text]
Hodgson, R., Bushe, C., Hunter, R.
(2007). Measurement of long-term outcomes in observational and randomised controlled trials. Br. J. Psychiatry
191: s78-s84
[Abstract][Full Text]
Boutron, I., Ravaud, P., Nizard, R.
(2007). The design and assessment of prospective randomised, controlled trials in orthopaedic surgery. J Bone Joint Surg Br
89-B: 858-863
[Abstract][Full Text]
Maeng, M., Okkels Jensen, L., Rasmussen, K., Flensted Lassen, J., Romer Krusell, L., Thayssen, P., Thuesen, L.
(2007). Target lesion revascularisation in patients treated with a sirolimus-eluting or paclitaxel-eluting stent. Heart
93: 694-697
[Abstract][Full Text]
The Society of Thoracic Surgeons Blood Conservatio, , Ferraris, V. A., Ferraris, S. P., Saha, S. P., Hessel, E. A. II, Haan, C. K., Royston, B. D., Bridges, C. R., Higgins, R. S.D., Despotis, G., Brown, J. R., The Society of Cardiovascular Anesthesiologists Sp, , Spiess, B. D., Shore-Lesserson, L., Stafford-Smith, M., Mazer, C. D., Bennett-Guerrero, E., Hill, S. E., Body, S.
(2007). Perioperative Blood Transfusion and Blood Conservation in Cardiac Surgery: The Society of Thoracic Surgeons and The Society of Cardiovascular Anesthesiologists Clinical Practice Guideline. Ann. Thorac. Surg.
83: S27-S86
[Abstract][Full Text]
Mangano, D. T., Miao, Y., Vuylsteke, A., Tudor, I. C., Juneja, R., Filipescu, D., Hoeft, A., Fontes, M. L., Hillel, Z., Ott, E., Titov, T., Dietzel, C., Levin, J., for the Investigators of The Multicenter Study of,
(2007). Mortality Associated With Aprotinin During 5 Years Following Coronary Artery Bypass Graft Surgery. JAMA
297: 471-479
[Abstract][Full Text]
Lucini, D., Riva, S., Pizzinelli, P., Pagani, M.
(2007). Stress Management at the Worksite: Reversal of Symptoms Profile and Cardiovascular Dysregulation. Hypertension
49: 291-297
[Abstract][Full Text]
Stukel, T. A., Fisher, E. S., Wennberg, D. E., Alter, D. A., Gottlieb, D. J., Vermeulen, M. J.
(2007). Analysis of Observational Studies in the Presence of Treatment Selection Bias: Effects of Invasive Cardiac Management on AMI Survival Using Propensity Score and Instrumental Variable Methods. JAMA
297: 278-285
[Abstract][Full Text]
DeBaun, M. R., Field, J. J.
(2007). Limitations of Clinical Trials in Sickle Cell Disease: A Case Study of the Multi-center Study of Hydroxyurea (MSH) Trial and the Stroke Prevention (STOP) Trial. ASH Education Book
2007: 482-488
[Abstract][Full Text]
Coulter, I. D.
(2006). Evidence Summaries and Synthesis: Necessary but Insufficient Approach for Determining Clinical Practice of Integrated Medicine?. Integr Cancer Ther
5: 282-286
[Abstract]
Bell, I. R., Koithan, M.
(2006). Models for the Study of Whole Systems. Integr Cancer Ther
5: 293-307
[Abstract]
Barrios, V., Escobar, C., Calderon, A., Pablo Tomas, J., Ruiz, S., Moya, J. L., Megias, A., Vegazo, O., Fernandez, R.
(2006). Regression of left ventricular hypertrophy by a candesartan-based regimen in clinical practice The VIPE study. Journal of Renin-Angiotensin-Aldosterone System
7: 236-242
[Abstract]
Reider, B.
(2006). More Random Thoughts. Am J Sports Med
34: 1889-1890
[Full Text]
Weinstein, J. N., Lurie, J. D., Tosteson, T. D., Skinner, J. S., Hanscom, B., Tosteson, A. N. A., Herkowitz, H., Fischgrund, J., Cammisa, F. P., Albert, T., Deyo, R. A.
(2006). Surgical vs Nonoperative Treatment for Lumbar Disk Herniation: The Spine Patient Outcomes Research Trial (SPORT) Observational Cohort. JAMA
296: 2451-2459
[Abstract][Full Text]
Bonovas, S., Filioussi, K., Tsavaris, N., Sitaras, N. M.
(2006). Statins and Cancer Risk: A Literature-Based Meta-Analysis and Meta-Regression Analysis of 35 Randomized Controlled Trials. JCO
24: 4808-4817
[Abstract][Full Text]
Hayward, R. A., Hofer, T. P., Vijan, S.
(2006). Narrative review: lack of evidence for recommended low-density lipoprotein treatment targets: a solvable problem.. ANN INTERN MED
145: 520-530
[Abstract][Full Text]
Sasson, L., Katz, M. G., Ezri, T., Tamir, A., Herman, A., Bove, E. L., Schachner, A.
(2006). Indications for tricuspid valve detachment in closure of ventricular septal defect in children.. Ann. Thorac. Surg.
82: 958-963
[Abstract][Full Text]
Trojano, M., Russo, P., Fuiani, A., Paolicelli, D., Di Monte, E., Granieri, E., Rosati, G., Savettieri, G., Comi, G., Livrea, P., MSDN Study Group,
(2006). The Italian Multiple Sclerosis Database Network (MSDN): the risk of worsening according to IFN{beta} exposure in multiple sclerosis. Mult Scler
12: 578-585
[Abstract]
Witt, C. M., Jena, S., Selim, D., Brinkhaus, B., Reinhold, T., Wruck, K., Liecker, B., Linde, K., Wegscheider, K., Willich, S. N.
(2006). Pragmatic Randomized Trial Evaluating the Clinical and Economic Effectiveness of Acupuncture for Chronic Low Back Pain. Am J Epidemiol
164: 487-496
[Abstract][Full Text]
Thadhani, R., Tonelli, M.
(2006). Cohort Studies: Marching Forward. CJASN
1: 1117-1123
[Full Text]
Tiihonen, J., Walhbeck, K., Lonnqvist, J., Klaukka, T., Ioannidis, J. P A, Volavka, J., Haukka, J.
(2006). Effectiveness of antipsychotic treatments in a nationwide cohort of patients in community care after first hospitalisation due to schizophrenia and schizoaffective disorder: observational follow-up study. BMJ
333: 224-
[Abstract][Full Text]
Ioannidis, J. P A
(2006). Commentary: Grading the credibility of molecular evidence for complex diseases. Int J Epidemiol
35: 572-578
[Full Text]
HAK, E., HOES, A. W, NORDIN, J., NICHOL, K. L
(2006). Benefits of influenza vaccine in US elderly--appreciating issues of confounding bias and precision. Int J Epidemiol
35: 800-802
[Full Text]
Hillner, B. E., Mandelblatt, J.
(2006). Caring for older women with breast cancer: can observational research fill the clinical trial gap?. JNCI J Natl Cancer Inst
98: 660-661
[Full Text]
Previous
