The Toolkit provides three ratings:

1. An impact rating

This describes whether, on average, the approach has had no effect, or a low, medium, high or harmful impact.

2. An evidence rating

This describes the confidence that we have in the research used to calculate the impact rating.

3. A cost rating

This gives a general indication of the cost of the approach, relative to other approaches in the Toolkit.

Each of the ratings is described in more detail below. A full, technical description can be found in the Toolkit Technical Guide.

Impact

The impact rating indicates the likely average impact of the approach on keeping children safe from involvement in violence. For example, the section on CBT shows that, on average, CBT is likely to have led to relatively large reductions in violence.

The rating is based on the average effect size reported in a meta-analysis. A meta-analysis is a type of study which aims to find as many studies as possible on a particular approach, and then calculate the average impact.

It’s important to remember that this rating refers to the average impact. The average is useful – it helps us to distinguish the approaches that are most likely to work. But there is still variation around this average. The detailed summaries describe what we know about what causes this variation, because it might help to design more effective activities. For example, if you click through to detailed summary on CBT, it tells you how programmes with more regular sessions have tended to have larger impacts.  

The Toolkit tries to estimate the average impact on violent crime but it is rare that studies directly measure the impact on violence. Research has tended to focus on more general measures of crime or things which predict violence. If we do not have a measure of the impact on violent crime, we use information about the impact on related outcomes (for example, reducing aggression or improving family relationships) to estimate the impact on violence.  

More detail on the impact rating is available in the Toolkit Technical Guide.

Evidence security

The evidence quality rating describes the confidence that we have in our impact rating. The possible evidence ratings range from one magnifying glass (very low confidence) to five magnifying glasses (very high confidence).

Where we have used existing systematic reviews, we use the following criteria to produce an evidence security rating:

• The quality of the systematic review that the impact rating is based on.
• The number of studies in that systematic review.
• The amount of variation in the results from the primary studies used to produce the impact rating.
• Whether the impact estimate is based on a direct measure of crime or violence,  or an indirect estimate based on an intermediate outcome such as bullying perpetration.

Where we have commissioned our own systematic reviews of the evidence (we will be updating each strand with our own review up to 2029), we use the following criteria to produce an evidence rating. This is different because we already know that the quality of the review is high, and that the review only includes studies involving children and young people:

• The number of studies.
• The quality of the studies included in producing the impact rating.
• If there is a lot of variation in the results, we knock down a rating if we can’t explain what caused the variation in impact.

What does each rating level mean? 

For more detail on how the evidence security rating is allocated see the Toolkit Technical Guide.

Cost

The Toolkit cost rating aims to provide an initial indication of the likely cost of implementing the approach. The rating places the cost of the approach in one of three bands: