Aims
• To estimate the prevalence of outcome reporting bias in trials within a cohort of published meta-analyses;
• To assess the impact of outcome reporting bias in a cohort of meta-analyses;
• To examine the prevalence and impact of outcome reporting bias in a cohort of individual patient data meta-analyses;
• To estimate the sensitivity and specificity of a method for assessing the likelihood of outcome reporting bias from trial reports;
• To compare the understanding of the research process between trialists who do and do not report all outcomes;
• To undertake a review of published and institutional guidelines on research conduct in relation to this issue.
Objectives and key findings
1) Review an unselected cohort of Cochrane systematic reviews to estimate the prevalence and impact of outcome reporting bias.
The main findings were that a third of Cochrane reviews contained at least one trial with high outcome reporting bias (ORB) suspicion for the review primary outcome. Considering meta-analyses with a statistically significant result only, just under a fifth were not robust to ORB and over a quarter would have overestimated the treatment effect by 20% or more. We found ORB to be an under-recognised problem which affects the inference in a substantial proportion of Cochrane reviews.
Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, Williamson PR. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ (2010); 340:c356.
2) Conduct interviews with trialists to understand the reasons for discrepancies between outcomes specified in the trial protocol and those reported in the study publication.
The main finding was that in over a quarter of the unselected trials, trialists were found to have reported the outcomes in a biased way. Trialists seemed generally unaware of the implications for the evidence base of not reporting all outcomes and protocol changes. Deliberate misrepresentation of the results appeared to be rare. A general lack of consensus regarding the importance of outcomes in particular clinical settings was evident and impacted on trial design, conduct, analysis and reporting.
Smyth R, Kirkham JJ, Jacoby A, Altman DG, Gamble C, Williamson, PR. Frequency and reasons for outcome reporting bias in clinical trials: interviews with trialists. BMJ (2011); 342:c7153.
3) Review trial funders’ guidelines for researchers to summarise how often, and in what way, outcome reporting bias and more broadly publication bias, are referred to.
The main findings were that only 15% of the guidelines referred to the importance of publishing negative as well as positive findings, none referred explicitly to the need to report results for all outcomes analysed, and that current guidelines need to be updated.
Dwan K, Gamble C, Williamson PR, Altman DG. Reporting of Clinical Trials: a review of research funders’ guidelines. Trials (2008); 9:66.
Back to: Institute of Population Health