Background Meta-analyses are now widely used to provide evidenceto support clinical strategies. However, large randomized, controlledtrials are considered the gold standard in evaluating the efficacyof clinical interventions.
Methods We compared the results of large randomized, controlledtrials (involving 1000 patients or more) that were publishedin four journals (the New England Journal of Medicine, the Lancet,the Annals of Internal Medicine, and the Journal of the AmericanMedical Association) with the results of meta-analyses publishedearlier on the same topics. Regarding the principal and secondaryoutcomes, we judged whether the findings of the randomized trialsagreed with those of the corresponding meta-analyses, and wedetermined whether the study results were positive (indicatingthat treatment improved the outcome) or negative (indicatingthat the outcome with treatment was the same or worse than withoutit) at the conventional level of statistical significance (P<0.05).
Results We identified 12 large randomized, controlled trialsand 19 meta-analyses addressing the same questions. For a totalof 40 primary and secondary outcomes, agreement between themeta-analyses and the large clinical trials was only fair (kappa= 0.35; 95 percent confidence interval, 0.06 to 0.64). The positivepredictive value of the meta-analyses was 68 percent, and thenegative predictive value 67 percent. However, the differencein point estimates between the randomized trials and the meta-analyseswas statistically significant for only 5 of the 40 comparisons(12 percent). Furthermore, in each case of disagreement a statisticallysignificant effect of treatment was found by one method, whereasno statistically significant effect was found by the other.
Conclusions The outcomes of the 12 large randomized, controlledtrials that we studied were not predicted accurately 35 percentof the time by the meta-analyses published previously on thesame topics.
Large randomized, controlled trials are generally consideredthe gold standard in evaluations of the efficacy of clinicalinterventions. However, since such trials are not always available,clinicians increasingly rely on meta-analysis to support theirchoice of clinical strategies. Critics have emphasized the intrinsicweaknesses of meta-analysis.1,2,3,4,5 Pooled results incorporatethe biases of individual studies and embody new sources of bias,mostly because of the selection of studies and the inevitableheterogeneity among them.
Although much has been said about the strengths and weaknessesof meta-analysis, there are limited data systematically comparingthe results of meta-analyses of several small trials with thoseof large randomized, controlled trials. Villar et al.6 reviewed30 meta-analyses of various interventions in perinatal medicinefrom the Cochrane data base. They recalculated the results ofeach meta-analysis after removing the largest trial from theanalysis and then compared the results with those of the largetrial that had been removed. They found a kappa of 0.46 to 0.53and a positive predictive value of 50 to 67 percent. We comparedthe results of a series of systematically compiled large randomized,controlled trials with those of the relevant meta-analyses thathad been published previously.
Methods
Data Base
We searched the New England Journal of Medicine, the Lancet,the Annals of Internal Medicine, and the Journal of the AmericanMedical Association and retrieved all large randomized, controlledtrials (those in which 1000 patients or more were studied) thatwere published between January 1, 1991, and December 31, 1994.All the trials had to have adequate statistical power to detectthe desired benefit specified by the authors. Adequate powerwas defined on the basis of the a priori calculations of powerreported by the authors in the Methods sections of their articles.We then searched for meta-analyses of similar topics that hadbeen published before the large randomized, controlled trial.Our search included the references listed in the randomizedtrials and computerized searches of Medline without languagerestrictions. We then compared each trial with the set of meta-analysescorresponding to it and selected only those meta-analyses thatcoincided with the trial with regard to the similarity of thepopulations studied, the therapeutic intervention, and at leastone outcome. We studied the principal and secondary outcomes.
For each outcome that was studied in both the large randomized,controlled trial and the meta-analysis, we determined whetherthe results were positive (indicating that treatment resultedin a better outcome) or negative (indicating that treatmentresulted in an equal or worse outcome) at the conventional levelof statistical significance (P<0.05). Two investigators workingindependently of each other reviewed each trial and its correspondingmeta-analyses. Discrepancies were resolved by consensus, withthe help of a third investigator. To quantify the effect ofinterobserver variation, we performed a sensitivity analysis;the statistical calculations were performed with the data obtainedby consensus and were repeated with the data that correspondedto the opinion of the dissenting investigator.
Statistical Analysis
Two-by-two tables were used to calculate the degree of agreementbetween the large randomized, controlled trial and its associatedmeta-analysis as expressed by the kappa statistic and its 95percent confidence interval, as well as the sensitivity, specificity,positive predictive value, and negative predictive value. Thepoint estimates in each pair were compared by using a test statisticconstructed as the difference in the proportions or means dividedby the square root of the sum of the variances.
The odds ratios of the randomized, controlled trial and themeta-analysis were represented graphically. When the resultof the meta-analysis was not presented as an odds ratio fora dichotomous outcome, we computed the odds ratio and its 95percent confidence interval by the fixed-effects Mantel–Haenszelmethod.7 When no odds ratio could be computed for a meta-analysisthat represented the size of the treatment effect, we transformedthe odds ratio in the corresponding randomized, controlled trialinto an effect size by treating the proportion for each groupas the mean of a distribution of 0's and 1's.8 These transformationswere made only to permit graphic representation and did notaffect the P values reported in the corresponding papers. Figure 1shows the odds ratios computed by the fixed-effects method,and Figure 2 shows the effect sizes obtained by transformationof the odds ratios. P values of less than 0.05 were consideredto indicate statistical significance. All the calculations andstatistical tests were done with the SAS statistical package(SAS Institute, Cary, N.C.).
Figure 1. Odds Ratios and 95 Percent Confidence Intervals for Clusters of Studies in Which the Findings of Large Randomized, Controlled Trials Were Compared with the Results of One or More Meta-Analyses on the Same Subject, in Which at Least One Common Outcome Was Studied.
Each randomized trial and its associated meta-analyses are separated from the others by a solid horizontal line. Dashed lines delineate each cluster of trials and meta-analyses in which a single outcome was examined. The solid squares at right are the point estimates (odds ratios) for the randomized trials, and the open squares are the odds ratios for the meta-analyses. The bars on either side of the squares are 95 percent confidence intervals.
The vertical line indicating an odds ratio of 1.0 is the line at which treatment was found to have no effect. Odds ratios to the left of that line (lower than 1.0) indicate that outcome was better with treatment; those to the right (higher than 1.0) indicate that outcome was worse. When the 95 percent confidence interval does not span the "no difference" line at 1.0, the study findings are considered to be significant (P<0.05).
Names of randomized, controlled trials are given in roman type, and names of meta-analyses are given in italics. For the randomized, controlled trials, the inclusive dates listed are the years when the first and last patients were enrolled; for the meta-analyses, the dates are the years when the first and last papers were published. Pub'd denotes published, ACE angiotensin-converting enzyme, and NS not specified.
Figure 2. Treatment Effects and 95 Percent Confidence Intervals after Transformation of the Odds Ratios in Clusters of Studies in Which the Findings of Large Randomized, Controlled Trials Were Compared with the Results of One or More Meta-Analyses on the Same Subject, in Which at Least One Common Outcome Was Studied.
Each randomized trial and its associated meta-analysis are separated from the others by a solid horizontal line. Dashed lines delineate each cluster of trials and meta-analyses in which a single outcome was examined. The solid squares at right are the point estimates (effect sizes) for the randomized trials, and the open squares are the effect sizes for the meta-analyses (calculated as described in the Methods section). The bars on either side of the squares are 95 percent confidence intervals. Arrows mean that the differences were not statistically significant but the 95 percent confidence intervals could not be determined.
The vertical line indicating an effect size of 0 is the line at which treatment was found to have no effect. Odds ratios to the left of that line (lower than 0) indicate that outcome was better with treatment; those to the right (higher than 0) indicate that outcome was worse. When the 95 percent confidence interval does not span the "no difference" line at 0, the study findings are considered to be significant (P<0.05).
Names of randomized, controlled trials are given in roman type, and names of meta-analyses are given in italics. For the randomized, controlled trials, the inclusive dates listed are the years when the first and last patients were enrolled; for the meta-analyses, the dates are the years when the first and last papers were published. NS denotes not specified, and pub'd published.
Results
We identified 12 large randomized, controlled trials to which19 meta-analyses corresponded in terms of the populations studied,the therapeutic interventions, and at least one outcome. Sinceboth the primary and the secondary outcomes were considered,a total of 40 outcomes coincided and were included in the analysis.
Table 1 shows the data on which we based our evaluation of theperformance of meta-analysis as a predictor of the results ofsubsequent large randomized, controlled trials. The meta-analysisoccupied the role usually assigned to a diagnostic test beingassessed, whereas the trial was considered the gold standard.Table 2 shows the results in terms of sensitivity, specificity,and negative and positive predictive values. The results forthe consensus opinion are all in a range of values (65 to 70percent) that corresponds to the values usually obtained inaverage diagnostic tests. The kappa statistic, which measuresagreement beyond that due to chance alone, was 0.35 (95 percentconfidence interval, 0.06 to 0.64). Kappa values at or below0.40 are considered to represent fair-to-slight agreement. Table 2also shows the results of the sensitivity analysis, whichcompares the results obtained when the calculations were madeon the basis of the consensus between investigators with thoseobtained when the calculations were based on the dissentinginvestigator's opinion.
Table 2. Variables Measuring the Ability of Meta-Analyses to Predict the Results of Large Randomized, Controlled Trials.
Figure 1 and Figure 2 show the results graphically and includethe most pertinent information about each cluster of comparisons.They show that independently of their statistical significance,the point estimates were on the same side of 1.0 in Figure 1and on the same side of 0 in Figure 2 in 32 of the 40 comparisons(80 percent). No situation was found in which the point estimateswere both statistically significant and on opposite sides of1.0 or 0. All the disagreements thus occurred because one resultshowed a statistically significant treatment effect, whereasthe other indicated that such an effect was lacking. There wasa statistically significant difference between the randomizedclinical trial and the meta-analysis in 5 of the 40 comparisons(12 percent).
Five positive outcomes from four meta-analyses10,28,31,37 thatused fixed-effects models were followed by negative randomizedclinical trials. We had the information needed to redo the statisticalanalyses with random-effects models for four of these outcomes,10,28,31and the results in all four remained statistically significant.
We found very good agreement between the meta-analysis and therandomized clinical trial with regard to the following six clinicalmatters: the effect of magnesium on overall mortality in patientswith myocardial infarction,12,13 the effect of treatment forhypercholesterolemia on coronary events and mortality from cardiovascularcauses among patients with coronary heart disease,14,15,16 theeffect of vitamin A supplementation on mortality from all causesand mortality from diarrhea among children in developing countries,18,19,20the effect of angiotensin-converting–enzyme inhibitorson the mortality of patients with congestive heart failure,21,22the effect of adjuvant therapy on disease-free survival in patientswith breast cancer,32,33 and the value of multiple interventionsas compared with single interventions in smoking cessation.38,39
Considerable divergence was evident in several other cases.With regard to the effects of late thrombolysis (thrombolysisperformed at least six hours after the first symptoms of myocardialinfarction)9,10,11 and nitroglycerin on mortality in patientswith myocardial infarction, the meta-analyses were positive,whereas the results of the subsequent large randomized, controlledtrials were on the positive side of 1.0 but were not statisticallysignificant. In these instances statistical power could nothave been the issue, because the randomized, controlled trialsincluded more patients than the meta-analyses. With regard tothe question of preventing intrauterine growth retardation withlow-dose aspirin in women at risk of preeclampsia, a clearlypositive meta-analysis28 with only 394 patients was followedby a very large randomized, controlled trial with 9364 patientsthat had negative results.27 Despite a negative meta-analysis,35a large randomized, controlled trial34 showed that sodium reductiondecreases diastolic blood pressure, whereas in the case of calciumsupplementation the reverse occurred.34,37
Since the decision to conduct a large randomized, controlledtrial could have been made when clinicians and researchers sawa meta-analysis as inconclusive, we examined whether the meta-analysishad already been published at the time the first patient wasrandomized in the corresponding clinical trial. Four of the12 trials9,21,30,38 had evidently been started and most probablydesigned after the publication of the corresponding meta-analysis.Of these four trials, two9,30 (evaluating the merits of thrombolysisand treatment with nitroglycerin) had results that divergedfrom those of the meta-analysis — that is, a negativerandomized, controlled trial did not confirm the findings ofa positive meta-analysis.
Discussion
Few will disagree with the use of the large randomized, controlledtrial as the gold standard in the evaluation of the efficacyof therapeutic interventions. All the meta-analyses except onethat were found by our process of systematic research had beenpublished in major peer-reviewed journals, where they were ina position to influence clinical practice.
The strategy we used to decide whether a given meta-analysiscorresponded to a specific randomized, controlled trial raisescertain methodologic issues. For the studies to qualify, thepopulation studied, the therapeutic intervention, and at leastone outcome had to be similar. In some cases, such similaritycould involve judgment and thus be subject to variation betweenobservers. By having two investigators decide independentlyon the appropriateness of each match, we could quantify thevariation and adjust for it. The sensitivity analysis (Table 2)shows that our findings were essentially the same both whenthe calculations were based on consensus and when they werebased on the opinion of the dissenting investigator. Anothermethodologic issue is raised by the dichotomous classificationof the results as positive or negative. The reason for choosingthis approach was that the outcome of interest was whether theresults of the meta-analysis should be applied to clinical practice.Clinical decisions tend to be dichotomous in that a treatmentis said either to work and be recommended or not to work andnot to be recommended.
According to our analysis, if there had been no subsequent randomized,controlled trial, the meta-analysis would have led to the adoptionof an ineffective treatment in 32 percent of cases (100 percentminus the positive predictive value) and to the rejection ofa useful treatment in 33 percent of cases (100 percent minusthe negative predictive value). It is important to recognizethat these measures of disagreement, which are constructed fromthe perspective of medical decision making, tend to overstatethe degree of statistical discrepancy. This is evident fromthe fact that in no case was there a divergence in which therandomized clinical trial and the meta-analysis gave statisticallysignificant and opposite answers. Furthermore, wherever thepoint estimates were located in relation to the "no difference"line, the difference in results between the meta-analysis andthe randomized, controlled trial was statistically significantfor only 5 of the 40 comparisons (12 percent); this does notappear to be a large percentage, since a divergence in 5 percentof cases would be expected on the basis of chance alone.
In our study, 46 percent of the divergences in results involveda positive meta-analysis followed by a negative randomized,controlled trial. There are several reasons why a meta-analysismight have positive results that would not be confirmed by asubsequent trial. Publication bias refers to the tendency ofinvestigators to preferentially submit studies with positiveresults for publication, and the tendency of editors to acceptthem. A meta-analysis that excluded unpublished studies or didnot locate and include them would thus be more likely to havea false positive result. The systematic exclusion of paperswritten in languages other than English (the "Tower of Babel"bias40) can add to the publication bias. In our sample, theuse of the fixed-effects model, which narrows the confidenceinterval, does not appear to account for the statistically positivemeta-analyses whose findings were not subsequently confirmedby a randomized trial, since the four studies that could bereanalyzed by the random-effects model remained positive andcontinued to have statistically significant results when thatreanalysis was done.
The remaining 54 percent of identified divergences involveda negative meta-analysis followed by a positive randomized,controlled trial. The heterogeneity of the trials included inthe meta-analysis may partially account for divergence of thistype, since meta-analysis assumes that such variation is mostlycaused by random error, rather than by differences in the characteristicsof the selected studies. A properly done meta-analysis involvesthe a priori determination of strict standards to ensure thatthe criteria used for the inclusion of patients, the administrationof the principal treatment, and the ascertainment of outcomeevents are similar in all the trials selected. Although accordingto these strict criteria the protocols of the selected trialslook very similar, their application usually yields very differentproducts. The patients enrolled in comparable trials may belongto the same basic population, but even small differences inthe criteria for diagnosis, coexisting conditions, severityof disease, and age will produce very different groups of patients.Differences in doses, time to onset, and duration of therapiescan also produce substantial disparity among trials that areincluded in meta-analyses with the intention of evaluating atherapeutic intervention. The choice of concomitant therapiesand the degree of leeway in their administration can also affectthe results. Changes in medical practice over time may alsoaccount for important differences in concomitant therapies,since the trials included in a given meta-analysis are oftenconducted over a period of a decade or more.
How should clinicians use meta-analyses, given that systematiccomparison with randomized clinical trials shows that they havepoor predictive ability? Most will agree that if a large, well-donerandomized trial has been conducted, practice guidelines shouldbe strongly influenced by its results. The question arises whenthe only available evidence is from a series of small randomized,controlled trials. The simplest solution, and currently themost popular one, has been to rely on the results of a meta-analysis.Our findings seem to indicate that summarizing all the informationcontained in a set of trials into a single odds ratio may greatlyoversimplify an extremely complex issue. The popularity of meta-analysismay at least partly come from the fact that it makes life simplerand easier for reviewers as well as readers. However, oversimplificationmay lead to inappropriate conclusions.
The result of this study would appear to encourage readers togo beyond the point estimates and confidence intervals thatrepresent the aggregate findings of a meta-analysis and, asCook et al. have suggested,41 look carefully at the studiesthat were included and evaluate the consistency of their results.When the results are mostly on the same side of the no-differenceline, the meta-analysis merits more confidence. Others may considerfollowing the advice of Horwitz42 and appraising each trialseparately. Although such an approach is admittedly more laborious,it has the advantage of allowing pragmatic clinicians to benefitfrom the diversity of studies by distinguishing the effectsof treatment among them.
We are indebted to Dr. Jean-François Boivin for helpfulcriticisms and to Ms. Hélène Harnois and Ms. AnitaMassicotte for clerical assistance.
Source Information
From the Research Center, Hôtel-Dieu de Montréal Hospital, and the Department of Medicine, Faculty of Medicine, University of Montreal — both in Montreal.
Address reprint requests to Dr. LeLorier at the Research Center, Hôtel-Dieu de Montréal, 3850 St. Urbain St., Pavilion Marie de la Ferre, 2nd Fl., Montreal, QC H2W 1T8, Canada.
References
Feinstein AR. Meta-analysis: statistical alchemy for the 21st century. J Clin Epidemiol 1995;48:71-79. [CrossRef][Medline]
Horwitz RI. "Large-scale randomized evidence: large, simple trials and overviews of trials": discussion: a clinician's perspective on meta-analyses. J Clin Epidemiol 1995;48:41-44. [CrossRef][Medline]
Bailar JC III. The practice of meta-analysis. J Clin Epidemiol 1995;48:149-157. [CrossRef][Medline]
Meinert CL. Meta-analysis: science or religion? Control Clin Trials 1989;10:Suppl:257S-263S. [Medline]
Abrahamson JH. Meta-analysis: a review of pros and cons. Public Health Rev 1990;18:1-47. [Medline]
Villar J, Carroli G, Belizán JM. Predictive ability of meta-analyses of randomised controlled trials. Lancet 1995;345:772-776. [CrossRef][Medline]
Mantel N, Haenszel W. Statistical aspects of the analysis of data from retrospective studies of disease. J Natl Cancer Inst 1959;22:719-748.
McNemar Q. Psychological statistics. 3rd ed. New York: John Wiley, 1962.
EMERAS (Estudio Multicéntrico Estreptoquinasa Repúblicas de América del Sur) Collaborative Group. Randomised trial of late thrombolysis in patients with suspected acute myocardial infarction. Lancet 1993;342:767-772. [CrossRef][Medline]
Yusuf S, Collins R, Peto R, et al. Intravenous and intracoronary fibrinolytic therapy in acute myocardial infarction: overview of results on mortality, reinfarction and side-effects from 33 randomized controlled trials. Eur Heart J 1985;6:556-585. [Free Full Text]
Yusuf S, Sleight P, Held P, McMahon S. Routine medical management of acute myocardial infarction: lessons from overviews of recent randomized controlled trials. Circulation 1990;82:Suppl II:II-117.
Woods KL, Fletcher S, Roffe C, Haider Y. Intravenous magnesium sulphate in suspected acute myocardial infarction: results of the second Leicester Intravenous Magnesium Intervention Trial (LIMIT-2). Lancet 1992;339:1553-1558. [CrossRef][Medline]
Teo KK, Yusuf S, Collins R, Held PH, Peto R. Effects of intravenous magnesium in suspected acute myocardial infarction: overview of randomised trials. BMJ 1991;303:1499-1503.
Randomised trial of cholesterol lowering in 4444 patients with coronary heart disease: the Scandinavian Simvastatin Survival Study (4S). Lancet 1994;344:1383-1389. [CrossRef][Medline]
Law MR, Wald NJ, Thompson SG. By how much and how quickly does reduction in serum cholesterol concentration lower risk of ischaemic heart disease? BMJ 1994;308:367-372. [Free Full Text]
Rossouw JE, Lewis B, Rifkind BM. The value of lowering cholesterol after myocardial infarction. N Engl J Med 1990;323:1112-1119. [Medline]
Ravnskov U. Cholesterol lowering trials in coronary heart disease: frequency of citation and outcome. BMJ 1992;305:15-19. [Erratum, BMJ 1992;305:505.]
Ghana VAST Study Team. Vitamin A supplementation in northern Ghana: effects on clinic attendances, hospital admissions, and child mortality. Lancet 1993;342:7-12. [Erratum, Lancet 1993;342:250.] [CrossRef][Medline]
Glasziou PP, Mackerras DE. Vitamin A supplementation in infectious diseases: a meta-analysis. BMJ 1993;306:366-370.
Fawzi WW, Chalmers TC, Herrera MG, Mosteller F. Vitamin A supplementation and child mortality: a meta-analysis. JAMA 1993;269:898-903. [Free Full Text]
The Acute Infarction Ramipril Efficacy (AIRE) Study Investigators. Effect of ramipril on mortality and morbidity of survivors of acute myocardial infarction with clinical evidence of heart failure. Lancet 1993;342:821-828. [Medline]
Furberg CD, Yusuf S. Effect of drug therapy on survival in chronic congestive heart failure. Am J Cardiol 1988;62:41A-45A. [Medline]
Kakkar VV, Cohen AT, Edmonson RA, et al. Low molecular weight versus standard heparin for prevention of venous thromboembolism after major abdominal surgery: the Thromboprophylaxis Collaborative Group. Lancet 1993;341:259-265. [CrossRef][Medline]
Nurmohamed MT, Rosendaal FR, Buller HR, et al. Low-molecular-weight heparin versus standard heparin in general and orthopaedic surgery: a meta-analysis. Lancet 1992;340:152-156. [CrossRef][Medline]
Leizorovicz A, Haugh MC, Chapuis FR, Samama MM, Boissel JP. Low molecular weight heparin in prevention of perioperative thrombosis. BMJ 1992;305:913-920.
Lassen MR, Borris LC, Christiansen HM, et al. Clinical trials with low molecular weight heparins in the prevention of postoperative thromboembolic complications: a meta-analysis. Semin Thromb Hemost 1991;17:Suppl 3:284-290.
CLASP (Collaborative Low-dose Aspirin Study in Pregnancy) Collaborative Group. CLASP: a randomised trial of low-dose aspirin for the prevention and treatment of pre-eclampsia among 9364 pregnant women. Lancet 1994;343:619-629. [CrossRef][Medline]
Imperiale TF, Petrulis AS. A meta-analysis of low-dose aspirin for the prevention of pregnancy-induced hypertensive disease. JAMA 1991;266:260-264. [Free Full Text]
The SOLVD Investigators. Effect of enalapril on survival in patients with reduced left ventricular ejection fractions and congestive heart failure. N Engl J Med 1991;325:293-302. [Abstract]
Gruppo Italiano per lo Studio della Sopravvivenza nell'Infarto Miocardico. GISSI-3: effects of lisinopril and transdermal glyceryl trinitrate singly and together on 6-week mortality and ventricular function after acute myocardial infarction. Lancet 1994;343:1115-1122. [Medline]
Yusuf S, Collins R, MacMahon S, Peto R. Effect of intravenous nitrates on mortality in acute myocardial infarction: an overview of the randomised trials. Lancet 1988;1:1088-1092. [Medline]
Wood WC, Budman DR, Korzun AH, et al. Dose and dose intensity of adjuvant chemotherapy for stage II, node-positive breast carcinoma. N Engl J Med 1994;330:1253-1259. [Erratum, N Engl J Med 1994;331:139.] [Free Full Text]
Hryniuk W, Levine MN. Analysis of dose intensity for adjuvant chemotherapy trials in stage II breast cancer. J Clin Oncol 1986;4:1162-1170. [Free Full Text]
The Trials of Hypertension Prevention Collaborative Research Group. The effects of nonpharmacologic interventions on blood pressure of persons with high normal levels: results of the Trials of Hypertension Prevention, phase I. JAMA 1992;267:1213-1220. [Erratum, JAMA 1992;267:2330.] [Free Full Text]
Cutler JA, Follmann D, Elliott P, Suh I. An overview of randomized trials of sodium reduction and blood pressure. Hypertension 1991;17:Suppl:I27-I33.
Whelton PK, Thaker GK, Klag MJ, Seidler AJ. Blood pressure effects of potassium supplementation. Circulation 1989;80:Suppl II:II-301.abstract
Cutler JA, Brittain E. Calcium and blood pressure: an epidemiologic perspective. Am J Hypertens 1990;3:137S-146S. [Medline]
Hollis JF, Lichtenstein E, Vogt TM, Stevens JV, Biglan A. Nurse-assisted counseling for smokers in primary care. Ann Intern Med 1993;118:521-525. [Free Full Text]
Kottke TE, Battista RN, DeFriese GH, Brekke ML. Attributes of successful smoking cessation interventions in medical practice: a meta-analysis of 39 controlled trials. JAMA 1988;259:2883-2889. [CrossRef][Medline]
Grégoire G, Derderian F, Le Lorier J. Selecting the language of the publications included in a meta-analysis: is there a Tower of Babel bias? J Clin Epidemiol 1995;48:159-163. [CrossRef][Medline]
Cook DJ, Guyatt GH, Laupacis A, Sackett DL, Goldberg RJ. Clinical recommendations using levels of evidence for antithrombotic agents. Chest 1995;108:Suppl:227S-230S. [Free Full Text]
Horwitz RI. Complexity and contradiction in clinical trial research. Am J Med 1987;82:498-510. [CrossRef][Medline]
Meta-Analyses and Large Randomized, Controlled Trials
Ioannidis J. P.A., Cappelleri J. C., Lau J., Bent S., Kerlikowske K., Grady D., Song F.-J., Sheldon T. A., Khan S., Williamson P., Sutton R., Stewart L. A., Parmar M. K.B., Tierney J. F., Sim I., Lavori P., Imperiale T. F., LeLorier J., Grégoire G., Bailar J. C.
Extract |
Full Text
N Engl J Med 1998;
338:59-62, Jan 1, 1998.
Correspondence
Piper, M. E., Smith, S. S., Schlam, T. R., Fiore, M. C., Jorenby, D. E., Fraser, D., Baker, T. B.
(2009). A Randomized Placebo-Controlled Clinical Trial of 5 Smoking Cessation Pharmacotherapies. Arch Gen Psychiatry
66: 1253-1262
[Abstract][Full Text]
Johnson, D. W.
(2009). Sevelamer versus calcium-based phosphate binders in chronic kidney disease: what should we conclude from the evidence to date?. Nephrol Dial Transplant
24: 2970-2972
[Full Text]
Piccini, J. P., Hasselblad, V., Peterson, E. D., Washam, J. B., Califf, R. M., Kong, D. F.
(2009). Comparative efficacy of dronedarone and amiodarone for the maintenance of sinus rhythm in patients with atrial fibrillation.. J Am Coll Cardiol
54: 1089-1095
[Abstract][Full Text]
Macfarlane, A. J. R., Prasad, G. A., Chan, V. W. S., Brull, R.
(2009). Does regional anaesthesia improve outcome after total hip arthroplasty? A systematic review. Br J Anaesth
103: 335-345
[Abstract][Full Text]
Doig, G. S., Simpson, F., Finfer, S., Delaney, A., Davies, A. R., Mitchell, I., Dobb, G., for the Nutrition Guidelines Investigators of the,
(2008). Effect of Evidence-Based Feeding Guidelines on Mortality of Critically Ill Adults: A Cluster Randomized Controlled Trial. JAMA
300: 2731-2741
[Abstract][Full Text]
Bleicher, A. G., Kanal, E.
(2008). Assessment of Adverse Reaction Rates to a Newly Approved MRI Contrast Agent: Review of 23,553 Administrations of Gadobenate Dimeglumine. Am. J. Roentgenol.
191: W307-W311
[Abstract][Full Text]
Messe, S. R., Kasner, S. E.
(2008). Patent Foramen Ovale in Cryptogenic Stroke: Not to Close. Circulation
118: 1999-2004
[Full Text]
Standley, J. M.
(2008). Does Music Instruction Help Children Learn to Read? Evidence of a Meta-Analysis. Update: Appliations of Research in Music Education
27: 17-32
[Abstract]
Popping, D. M., Elia, N., Marret, E., Remy, C., Tramer, M. R.
(2008). Protective Effects of Epidural Analgesia on Pulmonary Complications After Abdominal and Thoracic Surgery: A Meta-Analysis. Arch Surg
143: 990-999
[Abstract][Full Text]
Chida, Y., Steptoe, A.
(2008). Positive Psychological Well-Being and Mortality: A Quantitative Review of Prospective Observational Studies. Psychosom. Med.
70: 741-756
[Abstract][Full Text]
Stone, G. W.
(2008). Angioplasty Strategies in ST-Segment-Elevation Myocardial Infarction: Part I: Primary Percutaneous Coronary Intervention. Circulation
118: 538-551
[Full Text]
Gray, A., Goodacre, S., Newby, D. E., Masson, M., Sampson, F., Nicholl, J., the 3CPO Trialists,
(2008). Noninvasive Ventilation in Acute Cardiogenic Pulmonary Edema. NEJM
359: 142-151
[Abstract][Full Text]
MANDELL, B. F.
(2008). Meta-analyses, metaphysics, and reality. Cleveland Clinic Journal of Medicine
75: 400-400
[Full Text]
WALKER, E., HERNANDEZ, A. V., KATTAN, M. W.
(2008). Meta-analysis: Its strengths and limitations. Cleveland Clinic Journal of Medicine
75: 431-439
[Abstract][Full Text]
Tobin, M. J.
(2008). Rebuttal From Dr. Tobin. Chest
133: 1076-1077
[Full Text]
(2008). References. JOURNAL OF THE ICRU
8: 57-62
Ata, B., Urman, B.
(2008). Ultrasound guided embryo transfer does not offer any benefit in clinical outcome: a randomized controlled study. Hum Reprod
23: 457-458
[Full Text]
Sosna, J., Sella, T., Sy, O., Lavin, P. T., Eliahou, R., Fraifeld, S., Libson, E.
(2008). Critical Analysis of the Performance of Double-Contrast Barium Enema for Detecting Colorectal Polyps >= 6 mm in the Era of CT Colonography. Am. J. Roentgenol.
190: 374-385
[Abstract][Full Text]
Garg, A. X., Hackam, D., Tonelli, M.
(2008). Systematic Review and Meta-analysis: When One Study Is Just not Enough. CJASN
3: 253-260
[Full Text]
Ferraris, V. A., Edwards, F. H., Shahian, D. M., Ferraris, S. P.
(2008). Risk Stratification and Comorbidity. Card Surg Adult
3: 199-246
[Full Text]
Tonelli, M.
(2007). Vitamin D in Patients with Chronic Kidney Disease: Nothing New under the Sun. ANN INTERN MED
147: 880-881
[Full Text]
Hlatky, M. A, Bravata, D. M
(2007). Review: rosiglitazone increases risk of MI but does not differ from other drugs for CV death in type 2 diabetes. Evid. Based Med.
12: 169-170
[Full Text]
Gilbert, R. E.
(2007). Rosiglitazone: Opening Pandora's Black Box?. CJASN
2: 1329-1331
[Full Text]
Kereiakes, D. J.
(2007). The Emperor's New Clothes: Another Cypher Versus Taxus Post-Hoc Meta-Analysis. J Am Coll Cardiol
50: 1381-1385
[Full Text]
Shojania, K. G., Sampson, M., Ansari, M. T., Ji, J., Doucette, S., Moher, D.
(2007). How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis. ANN INTERN MED
147: 224-233
[Abstract][Full Text]
Cleophas, T. J., Zwinderman, A. H.
(2007). Meta-Analysis. Circulation
115: 2870-2875
[Full Text]
Kang, W., Kudsk, K. A.
(2007). Is There Evidence That the Gut Contributes to Mucosal Immunity in Humans?. JPEN J Parenter Enteral Nutr
31: 246-258
[Abstract][Full Text]
Purkayastha, S., Alkhamesi, N. A., Darzi, A. W.
(2007). Intraperitoneal Local Anesthesia During Laparoscopic Cholecystectomy: The Role of Meta-Analytical Subgroups and Delivery of the Local Anesthetic. Anesth. Analg.
104: 994-994
[Full Text]
Heyland, D. K., Dhaliwalm, R., Day, A., Drover, J., Cote, H., Wischmeyer, P.
(2007). Optimizing the Dose of Glutamine Dipeptides and Antioxidants in Critically Ill Patients: A Phase I Dose-Finding Study. JPEN J Parenter Enteral Nutr
31: 109-118
[Abstract][Full Text]
Reiter, D.
(2007). Comparing Apples to Oranges in Meta-analysis Studies--Reply. Arch Facial Plast Surg
9: 140-141
[Full Text]
Guzick, D. S.
(2007). Treating the Polycystic Ovary Syndrome the Old-Fashioned Way. NEJM
356: 622-624
[Full Text]
Connolly, S. S, Fitzpatrick, J. M
(2007). Medical treatment of benign prostatic hyperplasia. Postgrad. Med. J.
83: 73-78
[Abstract][Full Text]
Coulter, I. D.
(2006). Evidence Summaries and Synthesis: Necessary but Insufficient Approach for Determining Clinical Practice of Integrated Medicine?. Integr Cancer Ther
5: 282-286
[Abstract]
Ng, T. T., McGory, M. L., Ko, C. Y., Maggard, M. A.
(2006). Meta-analysis in Surgery: Methods and Limitations. Arch Surg
141: 1125-1130
[Abstract][Full Text]
Rossing, P., Parving, H.-H., de Zeeuw, D.
(2006). Renoprotection by blocking the RAAS in diabetic nephropathy--fact or fiction?. Nephrol Dial Transplant
21: 2354-2357
[Full Text]
Polin, R. A., Lorenz, J. M., Bateman, D. A.
(2006). Evidence-based Neonatology. NeoReviews
7: e474-e485
[Full Text]
Hollenberg, N. K., Epstein, M.
(2006). Renin Angiotensin System Blockade and Nephropathy: Why Is It Being Called into Question, and Should It Be?. CJASN
1: 1046-1048
[Full Text]
Gluud, L. L.
(2006). Bias in Clinical Intervention Research. Am J Epidemiol
163: 493-501
[Abstract][Full Text]
Maier, R. V.
(2006). What the surgeon of tomorrow needs to know about evidence-based surgery.. Arch Surg
141: 317-323
[Full Text]
Papanikolaou, P. N., Christidi, G. D., Ioannidis, J. P.A.
(2006). Comparison of evidence on harms of medical interventions in randomized and nonrandomized studies. CMAJ
174: 635-641
[Abstract][Full Text]
Haythornthwaite, J. A.
(2005). Clinical trials studying pharmacotherapy and psychological treatments alone and together. Neurology
65: S20-S31
[Abstract][Full Text]
Fraser, W. D., Hofmeyr, J., Lede, R., Faron, G., Alexander, S., Goffinet, F., Ohlsson, A., Goulet, C., Turcot-Lemay, L., Prendiville, W., Marcoux, S., Laperriere, L., Roy, C., Petrou, S., Xu, H.-R., Wei, B., the Amnioinfusion Trial Group,
(2005). Amnioinfusion for the Prevention of the Meconium Aspiration Syndrome. NEJM
353: 909-917
[Abstract][Full Text]
Ioannidis, J. P. A.
(2005). Contradicted and Initially Stronger Effects in Highly Cited Clinical Research. JAMA
294: 218-228
[Abstract][Full Text]
Hartling, L., McAlister, F. A., Rowe, B. H., Ezekowitz, J., Friesen, C., Klassen, T. P.
(2005). Challenges in Systematic Reviews of Therapeutic Devices and Procedures. ANN INTERN MED
142: 1100-1111
[Abstract][Full Text]
Devlin, J. W., Barletta, J. F.
(2005). Albumin for fluid resuscitation: Implications of the Saline Versus Albumin Fluid Evaluation. Am J Health Syst Pharm
62: 637-642
[Full Text]
Hlatky, M. A., Massie, B. M.
(2004). Cardiac Resynchronization for Heart Failure. ANN INTERN MED
141: 399-400
[Full Text]
Diamond, G. A., Kaul, S.
(2004). Prior convictions: bayesian approaches to the analysis and interpretation of clinical megatrials. J Am Coll Cardiol
43: 1929-1939
[Abstract][Full Text]
Furukawa, T.
(2004). Meta-analyses and megatrials: neither is the infallible, universal standard. Evid. Based Ment. Health
7: 34-35
[Full Text]
Kshirsagar, A. V., Poole, C., Mottl, A., Shoham, D., Franceschini, N., Tudor, G., Agrawal, M., Denu-Ciocca, C., Magnus Ohman, E., Finn, W. F.
(2004). N-Acetylcysteine for the Prevention of Radiocontrast Induced Nephropathy: A Meta-Analysis of Prospective Controlled Trials. J. Am. Soc. Nephrol.
15: 761-769
[Abstract][Full Text]
Ballantyne, J. C.
(2004). Editorial II: Does epidural analgesia improve surgical outcome?. Br J Anaesth
92: 4-6
[Full Text]
Jacka, M. J., Schricker, T., Warriner, B., Boulton, A., Hudson, R., Parlow, J. L., Milne, B., VanDenKerkhof, E. G.
(2004). More Conclusive Large-Scale Trials Necessary Before Recommending Use of Beta Blockade in Patients at Risk * Response. Anesth. Analg.
98: 269-270
[Full Text]
Sosna, J., Morrin, M. M., Kruskal, J. B., Lavin, P. T., Rosen, M. P., Raptopoulos, V.
(2003). CT Colonography of Colorectal Polyps: A Metaanalysis. Am. J. Roentgenol.
181: 1593-1598
[Abstract][Full Text]
McFadden, E. R. Jr.
(2003). Acute Severe Asthma. Am. J. Respir. Crit. Care Med.
168: 740-759
[Abstract][Full Text]
Bhandari, M., Devereaux, P. J., Swiontkowski, M. F., Tornetta, P. III, Obremskey, W., Koval, K. J., Nork, S., Sprague, S., Schemitsch, E. H., Guyatt, G. H.
(2003). Internal Fixation Compared with Arthroplasty for Displaced Fractures of the Femoral Neck: A Meta-Analysis. JBJS
85: 1673-1681
[Abstract][Full Text]
Bigby, M., Williams, H.
(2003). Appraising Systematic Reviews and Meta-analyses. Arch Dermatol
139: 795-798
[Abstract][Full Text]
Buyse, M., Burzykowski, T., Parmar, M., Torri, V., Omura, G., Colombo, N., Williams, C., Conte, P., Vermorken, J.
(2003). Using the Expected Survival to Explain Differences Between the Results of Randomized Trials: A Case in Advanced Ovarian Cancer. JCO
21: 1682-1687
[Abstract][Full Text]
Tobacman, J. K., Zimmerman, B., Lee, P., Hilborne, L., Kolder, H., Brook, R. H.
(2003). Visual Acuity following Cataract Surgeries in Relation to Preoperative Appropriateness Ratings. Med Decis Making
23: 122-130
[Abstract]
Peyton, P. J., Myles, P. S., Silbert, B. S., Rigg, J. A., Jamrozik, K., Parsons, a. R.
(2003). Perioperative Epidural Analgesia and Outcome After Major Abdominal Surgery in High-Risk Patients. Anesth. Analg.
96: 548-54
[Abstract][Full Text]
Ferraris, V. A., Ferraris, S. P.
(2003). Risk Stratification and Comorbidity. Card Surg Adult
2: 187-224
[Full Text]
Goodman, N. W., Pronovost, P., Berenholtz, S., Dorman, T., Martinez, E., Merritt, W.
(2002). Evidence-Based Medicine Needs Proper Critical Review * Response. Anesth. Analg.
95: 1817-1818
[Full Text]
Buchholz, T. A., Strom, E. A., Perkins, G. H., McNeese, M. D.
(2002). Controversies Regarding the Use of Radiation After Mastectomy in Breast Cancer. The Oncologist
7: 539-546
[Abstract][Full Text]
Thabut, G., Thabut, D., Myers, R. P., Bernard-Chabert, B., Marrash-Chahla, R., Mal, H., Fournier, M.
(2002). Thrombolytic therapy of pulmonary embolism: A meta-analysis. J Am Coll Cardiol
40: 1660-1667
[Abstract][Full Text]
Raja, S. N., Haythornthwaite, J. A., Pappagallo, M., Clark, M. R., Travison, T. G., Sabeen, S., Royall, R. M., Max, M. B.
(2002). Opioids versus antidepressants in postherpetic neuralgia: A randomized, placebo-controlled trial. Neurology
59: 1015-1021
[Abstract][Full Text]
STIMPSON, N., AGRAWAL, N., LEWIS, G.
(2002). Randomised controlled trials investigating pharmacological and psychological interventions for treatment-refractory depression: Systematic review. Br. J. Psychiatry
181: 284-294
[Abstract][Full Text]
Hachinski, V.
(2002). Cochrane Corner. Stroke
33: 1739-1739
[Full Text]
Bonow, R. O.
(2002). Myocardial viability and prognosis in patients with ischemic left ventricular dysfunction. J Am Coll Cardiol
39: 1159-1162
[Full Text]
Juni, P., Holenstein, F., Sterne, J., Bartlett, C., Egger, M.
(2002). Direction and impact of language bias in meta-analyses of controlled trials: empirical study. Int J Epidemiol
31: 115-123
[Abstract][Full Text]
Kjaergard, L. L., Villumsen, J., Gluud, C.
(2001). Reported Methodologic Quality and Discrepancies between Large and Small Randomized Trials in Meta-Analyses. ANN INTERN MED
135: 982-989
[Abstract][Full Text]
Eisenberg, M. J.
(2001). Risk stratification for arrhythmic events: are the bangs worth the bucks?. J Am Coll Cardiol
38: 1912-1915
[Full Text]
Contopoulos-Ioannidis, D. G., Ioannidis, J. P. A., Chew, P., Lau, J.
(2001). Meta-analysis of randomized controlled trials on the comparative efficacy and safety of azithromycin against other antibiotics for lower respiratory tract infections. J Antimicrob Chemother
48: 691-703
[Abstract][Full Text]
CAPLAN, L R
(2001). Evidence based medicine: concerns of a clinical neurologist. J. Neurol. Neurosurg. Psychiatry
71: 569-574
[Full Text]
Glazer, W. M., Woods, S. W., Goff, D.
(2001). Should Sisyphus Have Taken Melatonin?. Arch Gen Psychiatry
58: 1054-1055
[Full Text]
Inglis, A., Price, R., Bion, J., O'Leary, M. J, Bihari, D. J
(2001). Mortality after discharge from intensive care. BMJ
323: 629-629
[Full Text]
Lipsey, M. W., Wilson, D. B.
(2001). The Way in Which Intervention Studies Have "Personality" and why it is Important to Meta-Analysis. Eval Health Prof
24: 236-254
[Abstract]
Sobel, B. E., Levine, M. A.
(2001). Medical Education, Evidence-Based Medicine, and the Disqualification of Physician-Scientists. Exp. Biol. Med.
226: 713-716
[Full Text]
Song, F., Sheldon, T. A., Sutton, A. J., Abrams, K. R., Jones, D. R.
(2001). Methods for Exploring Heterogeneity in Meta-Analysis. Eval Health Prof
24: 126-151
[Abstract]
BURROUGHS, A K, PATCH, D W, JALAN, R, HAYES, P C
(2001). Management of variceal haemorrhage in cirrhotic patients Reply. Gut
48: 738a-740
[Full Text]
Brophy, J. M., Joseph, L., Rouleau, J. L.
(2001). {beta}-Blockers in Congestive Heart Failure: A Bayesian Meta-Analysis. ANN INTERN MED
134: 550-560
[Abstract][Full Text]
Eikelboom, J. W., Mehta, S. R., Pogue, J., Yusuf, S.
(2001). Safety Outcomes in Meta-analyses of Phase 2 vs Phase 3 Randomized Trials: Intracranial Hemorrhage in Trials of Bolus Thrombolytic Therapy. JAMA
285: 444-450
[Abstract][Full Text]
Messori, A, Trippoli, S, Vaiani, M, Gorini, M, Corrado, A
(2000). Bleeding and pneumonia in intensive care patients given ranitidine and sucralfate for prevention of stress ulcer: meta-analysis of randomised controlled trials. BMJ
321: 1103-1103
[Abstract][Full Text]
Freeman, B. D., Isabella, K., Lin, N., Buchman, T. G.
(2000). A Meta-analysis of Prospective Trials Comparing Percutaneous and Surgical Tracheostomy in Critically Ill Patients. Chest
118: 1412-1418
[Abstract][Full Text]
Constantinescu, C S, Whiteley, A, Blumhardt, L D
(2000). Long term azathioprine fails to prevent onset of multiple sclerosis: report of two cases. Mult Scler
6: 362-363
[Abstract]
Arrivé, L., Renard, R., Carrat, F., Belkacem, A., Dahan, H., Le Hir, P., Monnier-Cholley, L., Tubiana, J.-M.
(2000). A Scale of Methodological Quality for Clinical Studies of Radiologic Examinations. Radiology
217: 69-74
[Abstract][Full Text]
Wu, C. L., Fleisher, L. A.
(2000). Outcomes Research in Regional Anesthesia and Analgesia. Anesth. Analg.
91: 1232-1242
[Full Text]
Witorsch, R. J., Witorsch, P.
(2000). Review : Environmental Tobacco Smoke and Respiratory Health in Children: A Critical Review and Analysis of the Literature from 1969 to 19981. Indoor and Built Environment
9: 246-264
[Abstract]
Arriagada, R., Pignon, J.-P.
(2000). Is Meta-analysis a Metaphysical or a Scientific Method?. Chest
118: 832-834
[Full Text]
Kubler, W
(2000). Treatment of cardiac diseases: evidence based or experience based medicine?. Heart
84: 134-136
[Full Text]
Concato, J., Shah, N., Horwitz, R. I.
(2000). Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs. NEJM
342: 1887-1892
[Abstract][Full Text]
Alsarraf, R., Alsarraf, N. W., Kato, B. M., Goldman, N. D.
(2000). Meta-analysis in Otolaryngology. Arch Otolaryngol Head Neck Surg
126: 711-716
[Abstract][Full Text]
Cheson, B. D.
(2000). The Curious Case of the Baffling Biological. JCO
18: 2007-2009
[Full Text]
Froom, P.
(2000). Blood viscosity and th erisk of death from coronary heart disease. Eur Heart J
21: 513-514
Barnett, H. J. M., Eliasziw, M.
(2000). Aspirin Benefit Remains Elusive in Primary Stroke Prevention. Arch Neurol
57: 306-308
[Full Text]
Hess, D. C., Demchuk, A. M., Brass, L. M., Yatsu, F. M.
(2000). HMG-CoA reductase inhibitors (statins): A promising approach to stroke prevention. Neurology
54: 790-796
[Abstract][Full Text]
Agerholm-Larsen, B., Nordestgaard, B. G., Tybjarg-Hansen, A.
(2000). ACE Gene Polymorphism in Cardiovascular Disease : Meta-Analyses of Small and Large Studies in Whites. Arterioscler. Thromb. Vasc. Bio.
20: 484-492
[Abstract][Full Text]
Spring, D. B., Barkan, H. E., Pruyn, S. C.
(2000). Potential Therapeutic Effects of Contrast Materials in Hysterosalpingography: A Prospective Randomized Clinical Trial1. Radiology
214: 53-57
[Abstract][Full Text]
Kong, D. F., Topol, E. J., Bittl, J. A., White, H. D., Theroux, P., Hasselblad, V., Califf, R. M.
(1999). Clinical Outcomes of Bivalirudin for Ischemic Heart Disease. Circulation
100: 2049-2053
[Abstract][Full Text]
Previous
