The evidence standards will help us gain a deeper understanding of what is working, when, for whom and under what conditions. They are an essential way of gaining a common understanding of 'what works', and more importantly, ensuring that this can be acted upon. We are applying standards of evidence to each of our maps, tools, and studies.
Each intervention in the tool has a rating for strength of evidence, cost effectiveness and impact. These ratings represent and aggregate of the studies that have been conducted for that intervention. The thresholds for the ratings are as follows:
Considerable reliable evidence:
At least 3 RCTs or 5 other studies with combined sample size of at least 300
Some reliable evidence:
At least 2 RCTs or 3 other studies with combined sample size of at least 200
Limited reliable evidence:
At least 1 RCT or 2 other studies with combined sample size of 100
Insuffienct evidence available:
Any other number or combination of studies.
To be determined
++
3 or more RCTs showing positive effect. No studies showing adverse effect.
+
At least one RCT or 2 other studies showing positive effect and no study showing adverse effect.
+/-
At least 2 RCTs showing insigificant effects.
–
At least one RCT or 2 other studies showing negative effect and no study showing positive effect.
––
3 or more RCTs showing negative effect. No studies showing positive effect.
N/A
Insufficient evidence available or any other number or combination of studies and findings.
These are interim standards - longer term we are working towards the standards recommended in this report
Each study in the map has been rated as high, medium or low for ‘confidence in study findings’. For systematic reviews in the map this rating was made using the revised version of ‘A MeaSurement Tool to Assess systematic Reviews’ (AMSTAR 2). The rating of primary studies was made using a critical appraisal tool based on various approaches to risk of bias assessment.
The two tools – AMSTAR 2 and the primary study critical appraisal tool – assess a range of items regarding study design and reporting. Some of these items are designated as ‘critical’. The overall rating for the study is the lowest rating or any critical item.
Study design: We have high confidence in study findings from studies designs best able to detect causal effects such as randomized control trials.
Attrition: High levels of attrition, especially differential attrition between the treatment and comparison groups, reduce the confidence we can have in study findings.
Outcome measure: For the study findings to be usable and meaningful there should be a clear description of the outcome measures, preferably using existing, validated approaches.
Baseline balance: We can have less confidence in study findings if there were significant differences between the treatment and comparison groups at baseline.
(assessed but not affecting overall rating):
Blinding: The absence of blinding of participants and researchers can bias study findings. This may be so even though blinding is not possible.
Power calculations: Power calculations help determine the sample size required. Without such calculations there is a risk of underpowered studies and so a high likelihood of not correctly identifying effective programmes.
Description of intervention: A clear description of the intervention is necessary to be clear what is being evaluated, so that effectiveness is not assigned to similar, but different, interventions.
Protocol registered before commencement of the review
Adequacy of the literature search.Justification for excluding individual studies
Risk of bias from individual studies being included in the review
Appropriateness of meta-analytical methods
Consideration of risk of bias when interpreting the results of the review
Assessment of presence and likely impact of publication bias
PICOS in inclusion criteria
Rationale for included study designs
Duplicate screening
Duplicate data extraction
Adequate description of included studies
Report sources of funding
Risk of bias assessment for meta-analysis
Analysis of heterogeneity
Report conflicts of interest
For a complete description of the coding tools click here for the primary study critical appraisal tool and here for AMSTAR 2.
