Bias in systematic reviews can arise from results that are reported and not reported. The former was described, and addressed for a particular setting, in Hutton and Williamson (2000). Reported results can lead to bias when an outcome is selectively reported based on a subset of the analysis undertaken. For example, a study may measure several outcomes related to a particular domain but only some of these outcomes (subscales) are reported. Incomplete reporting of outcomes is not limited to subscales, it can also occur when an arbitrary cut-off is selected for dichotomising a continuous measure or when a specific time point on which to focus has been chosen when the same outcome has been measured at multiple time points. The problem may equally apply to subgroups, prognostic factors and first period results in crossover trials. Non-reporting bias occurs when it is either clear or assumed that the outcome had been measured and possible that non-reporting could have been influenced by the results.
The text below addresses how to identify non-reporting bias in a systematic review.
Tools for identifying missing outcome data at the study level in a review (Outcome Matrix)
To be considered a reliable source of evidence about healthcare practice systematic reviewers should explicitly address the issue of missing outcome data at the study level. We have proposed an outcome matrix be used for this purpose (Kirkham et al, 2010). This outcome matrix has already been successfully applied and some examples of verbatim text from Cochrane Reviews and Protocols using our outcome matrix methodology include:
“To assess outcome reporting bias we will complete a review outcome matrix following the ORBIT classification” (Ang et al, 2010).
“We investigated selective outcome reporting by doing an “outcome matrix” and classifying missing outcomes according to the ORBIT classification” (Evans et al, 2010).
In the webpage below we outline five steps for addressing missing outcome data in a review.
Step 1 - Exclusion Criteria
The first step is to ensure that no potentially eligible studies are excluded from the review for the sole reason of not reporting on any review outcomes of interest. If a study report does not give results for, or mention, certain outcomes this does not necessarily mean that they were not measured or analysed. For this reason, studies MUST NOT be excluded if they do not report on any of the relevant review outcomes.
Step -2 Constructing the Outcome Matrix from study reports
The outcome matrix is constructed by listing all the eligible studies as rows and all the review outcomes of interest as columns in the matrix. Outcomes can be distinguished in terms of review primary and secondary outcomes. Outcomes that are not of interest in the review but are reported in the reports for eligible trials are also listed. This has been found to be useful when assigning the risk of bias. For example, if cause-specific mortality had been reported then all-cause mortality must have also been noted. Also, some outcomes are often measured routinely together so that if one outcome is reported, but not the other, this may raise suspicion that selective reporting has occurred e.g. systolic and diastolic blood pressure.
An example of a blank outcome matrix is provided using the following link (Blank Outcome Matrix)
Please feel free to use our ORBIT Matrix Generator which will automatically create your blank matrix.
Step 3 - Completing the Outcome Matrix
Once the outcome matrix has been constructed, the outcome matrix can be filled in. Here we provide some guidance for completing the outcome matrix. The matrix can also be filled in using the ORBIT matrix generator.
For each study, you should indicate which outcomes were reported and differentiate between ‘full reporting’, ‘partial reporting’, ‘not reported – not clear whether measured or not’ and ‘not measured’.
Full Reporting
Studies that fully report an outcome should be denoted by a tick. Typically if a study includes enough information on the outcome to be included in a review meta-analysis (without contact for extra information from the trialist) then the outcome is fully reported and there is no risk of non-reporting bias. If a meta-analysis is not appropriate then the outcome can still be considered fully reported. Some guidance on what constitutes full reporting for each of the different outcome types (e.g. binary, continuous and time to event) is provided in Section 7.7 (Extracting study results and converting to the desired format) of the Cochrane Handbook (Higgins & Deeks, 2011).
Partial Reporting
Studies that partially report an outcome should be denoted by an open circle. For partial reporting it is clear that the review outcome of interest has been analysed in the eligible studies but there is not enough information reported for the outcome to be considered fully reported (see above). Here are some tips for classifying outcomes as partially reported:
• P-value reported only (with no treatment effect size indicated) (high risk of bias).
• No measure of variance/precision reported (low risk of bias).
• Percentage data given for both groups but denominators unclear (binary data) (low risk of bias).
• Results are reported graphically only (accurate data extraction not possible) (low risk of bias).
• *Treatment effect estimate reported from a statistical model only (low risk of bias).
* This would be accepted as full reporting if the standard error of the estimate is also given.
Not reported – not clear whether measured or not
Studies that do not report on an outcome should be denoted by a cross. Reviewers should compare the methods to the results section of the study report, by looking at which other outcomes were measured and reported and accounting for knowledge of the clinical area. Reviewers should be suspicious of high risk of bias, if it is either clear or assumed that the outcome had been measured and possible that non-reporting could have been influenced by the results.
Not measured
On occasion it will be clear from the study reports that outcomes were not measured, these should be denoted by a star. If it is clear from the study report that outcomes were not measured then this eliminates any risk of non-reporting bias. Examples of outcomes that were clearly not measured in study reports found in the ORBIT study are provided:
Review outcome – Low back pain: “No data were gathered concerning the effect of cold laser therapy on pain” (Snyder et al, 1986).
Review outcome– Muscle strength: “No measurements of muscle strength were taken because the assessment of muscle strength with hemiparetic subjects is very difficult” (Mulder et al, 1986).
Review outcome– Type 2 diabetes: “Four weeks were not long enough to assess a long-term process like the development of glucose intolerance and diabetes” (Marreiro, 2002).
An Example of a Completed Matrix
An example of a completed outcome matrix is presented below. This particular example was from a review entitled “Therapeutic interventions for Burkitt lymphoma in children” (Okebe et al, 2006) and was one of the reviews included in the cohort of Cochrane systematic reviews used to estimate the prevalence and impact of outcome reporting bias (Kirkham et al, 2010).
Step 4 - Contacting Trialists
After the outcome matrix has been completed, reviewers should make an attempt to contact the trialists from the studies included in the review that partially reported the review outcomes of interest or where it was not clear whether the outcome was measured or not. The purpose of this contact is to try and obtain missing outcome data to include in the review analysis or to confirm that the outcomes of interest were not measured. The matrix should be updated accordingly.
Step 5 - Assessment of ORB
Once the outcome matrix is complete and trialists have been contacted for missing data, a reviewer may then want to assess the potential risk of outcome reporting bias as a result of outcomes being partially reported or measured but not reported. A tutorial for assessing the potential for outcome reporting bias in a review, using the ORBIT classification system (Kirkham et al, 2010) is provided in Dwan et al (2010) [http://www.trialsjournal.com/content/pdf/1745-6215-11-52.pdf].
Go to ORBIT Matrix Generator
References
Ang M, Mehta JS, Evans JR. Extracapsular cataract extraction (ECCE) with posterior chamber intraocular lens versus manual small incision cataract surgery (MSICS) with posterior chamber intraocular lens for age-related cataract (Protocol). Cochrane Database of Systematic Reviews 2010, Issue 11. Art. No.: CD008811. DOI: 10.1002/14651858.CD008811.
Dwan K, Gamble C, Kolamunnage-Dona R, Mohammed S, Powell C, Williamson PR. Assessing the potential for outcome reporting bias in a review: A tutorial. Trials (2010); 11:52.
Evans JR, Sivagnanavel V, Chong V. Radiotherapy for neovascular age-related macular degeneration. Cochrane Database of Systematic Reviews 2010, Issue 5. Art. No.: CD004004. DOI: 10.1002/14651858.CD004004.pub3.
Higgins JPT, Deeks JJ (editors). Chapter 7: Selecting studies and collecting data. In: Higgins JPT, Green S (editors), Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org.
Hutton JL, Williamson PR. Bias in meta-analysis due to outcome variable selection within studies. Journal of the Royal Statistical Society Series C (2000); 49: 359-370.
Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, Williamson PR. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ (2010); 340:c356.
Marreiro DN. The effect of zinc supplementation in insulin resistance in obesity women. [Efeito da suplementação com zinco na resistência à insulina em mulheres obesas]. Tese de Doutorado - Faculdade de Ciências Farmacêuticas - USP, São Paulo, Brazil. São Paulo, 2002:109p
Mulder T, Hulstijn W and van der Meer J. EMG feedback and the restoration of motor control: a controlled group study of 12 hemiparetic patients. American Journal of Physical Medicine 1986;65(4):173-88
Okebe JU, Skoetz N, Meremikwu MM, Richards S. Therapeutic interventions for Burkitt lymphoma in children. Cochrane Database of Systematic Reviews 2011, Issue 7. Art. No.: CD005198. DOI: 10.1002/14651858.CD005198.pub3.
Snyder ML, Bork C, Bourbon B and Trumbore D. Effect of helium-neon laser on musculoskeletal trigger points. Physical Therapy 1986;66:1087-90
Back to: Institute of Population Health