Our ApproachDownloads & ResourcesStandardsEmerging Leaders Programme

Standards.

We need standards of evidence to have a consistent way of assessing the evidence we use and generate.

The evidence standards will help us gain a deeper understanding of what is working, when, for whom and under what conditions. They are an essential way of gaining a common understanding of 'what works', and more importantly, ensuring that this can be acted upon. We are applying standards of evidence to each of our maps, tools, and studies.

Why standards of evidence matter

A vital first step
Standards of evidence will underpin all the centre’s projects. They will provide a consistent and transparent way of assessing the evidence we generate and use.

Helps us compare
We can compare different interventions, to understand how effective they are, how much they cost, and how confident we can be in what the evidence says.

Common language
Provide a common way of talking about evidence, to help us understand evidence, impact, and different evaluation methods.

Clarity
Ensure clarity about what evidence is, the quality of evidence, and how confident we can be in what it says.

Start conversations 
Standards of evidence are not a test of pass or fail, but a framework to understand how well an intervention is working now, and what our next steps are to deepen our knowledge of it.

Shows us the gaps
Highlight where there are gaps in our knowledge. No research is an important finding: it can help shape future research agendas.

Evidence standards

We are applying standards of evidence to each of our tools and maps, which have been developed with our partners The Campbell Collaboration.

Standards of Evidence for the Intervention Tool

Each intervention in the tool has a rating for strength of evidence, cost effectiveness and impact. These ratings represent and aggregate of the studies that have been conducted for that intervention. The thresholds for the ratings are as follows:

STRENGTH OF EVIDENCE

Considerable reliable evidence: 

At least 3 RCTs or 5 other studies with combined sample size of at least 300

Some reliable evidence: 

At least 2 RCTs or 3 other studies with combined sample size of at least 200

Limited reliable evidence: 

At least 1 RCT or 2 other studies with combined sample size of 100

Insuffienct evidence available:

Any other number or combination of studies.

COST EFFECTIVENESS

To be determined

IMPACT

++

3 or more RCTs showing positive effect. No studies showing adverse effect.

+

At least one RCT or 2 other studies showing positive effect and no study showing adverse effect.

+/-

At least 2 RCTs showing insigificant effects.

At least one RCT or 2 other studies showing negative effect and no study showing positive effect.

––

3 or more RCTs showing negative effect. No studies showing positive effect.

N/A

Insufficient evidence available or any other number or combination of studies and findings.

These are interim standards - longer term we are working towards the standards recommended in this report

Evidence Standards

The Centre is  applying standards of evidence to each of our tools and maps, which have been developed with our partners The Campbell Collaboration.

Standards of Evidence for the Effectiveness Map

Each study in the map has been rated as high, medium or low for ‘confidence in study findings’.  For systematic reviews in the map this rating was made using the revised version of ‘A MeaSurement Tool to Assess systematic Reviews’ (AMSTAR 2).  The rating of primary studies was made using a critical appraisal tool based on various approaches to risk of bias assessment.

The two tools – AMSTAR 2 and the primary study critical appraisal tool – assess a range of items regarding study design and reporting. Some of these items are designated as ‘critical’. The overall  rating for the study is the lowest rating or any critical item.

Primary studies

Critical items


Study design:   We have high confidence in study findings from studies designs best able to detect causal effects such as randomized control trials.


Attrition: High levels of attrition, especially differential attrition between the treatment and comparison groups, reduce the confidence we can have in study findings.


Outcome measure:  For the study findings to be usable and meaningful there should be a clear description of the outcome measures, preferably using existing, validated approaches.


Baseline balance: We can have less confidence in study findings if there were significant differences between the treatment and comparison groups at baseline.


Other items

(assessed but not affecting overall rating):


Blinding:  The absence of blinding of participants and researchers can bias study findings. This may be so even though blinding is not possible.


Power calculations: Power calculations help determine the sample size required. Without such calculations there is a risk of underpowered studies and so a high likelihood of not correctly identifying effective programmes.


Description of intervention: A clear description of the intervention is necessary to be clear what is being evaluated, so that effectiveness is not assigned to similar, but different, interventions.

Systematic Reviews

Critical items

Protocol registered before commencement of the review

Adequacy of the literature search.Justification for excluding individual studies

Risk of bias from individual studies being included in the review

Appropriateness of meta-analytical methods

Consideration of risk of bias when interpreting the results of the review  

Assessment of presence and likely impact of publication bias

Other items

PICOS in inclusion criteria

Rationale for included study designs

Duplicate screening

Duplicate data extraction

Adequate description of included studies

Report sources of funding

Risk of bias assessment for meta-analysis

Analysis of heterogeneity

Report conflicts of interest

For a complete description of the coding tools click here for the primary study critical appraisal tool and here for AMSTAR 2.