Evaluation type
What does this project involve?
This project aimed to test the feasibility of running a Multi-Site Trial (MST) as a method to robustly evaluate the provision of smaller youth organisations. Specifically, it aimed to test the process of developing and delivering a shared practice model of their mentoring provision.
Why did YEF fund this project?
Most rigorous impact evaluations of youth sector provision focus on well-defined, manualised programmes delivered by single organisations. These evaluations require large sample sizes that are often only achievable by large organisations. This is likely to exclude small, community-led, ‘grass-roots’ organisations from participating in evaluations, and from testing the feasibility and efficacy of their interventions. Smaller organisations may be more likely to be led by, and serve, individuals from racially minoritised communities, and they may also provide specialist, more adaptable services not offered by larger organisations. Due to the complexity, cost, training time, and rigidity of some programmes, smaller organisations may not be best placed to deliver wide-spread manualised interventions. In addition, most services provided to young people in England and Wales take place in small, local organisations and consist of non-manualised, informal approaches to support. This creates a gap between the available evidence and everyday practice.
This project aimed to test an approach to closing this gap, by examining the feasibility of running a Multi-Site Trial (MST) with smaller organisations. An MST aims to combine the reach of multiple smaller organisations, to create a sample large enough for an impact evaluation. To test this in this project, smaller organisations (Delivery Partner Organisations or ‘DPOs’) delivered a common and promising provision: mentoring.
Mentoring programmes match a child with a mentor and encourage them to meet regularly. They aim to help the child form a good relationship with a trusted adult, and develop social skills, form constructive relationships with others, and develop positive behaviours and aspirations. Many mentoring programmes also specifically aim to reduce involvement in violence. This project aimed to establish the feasibility of engaging DPOs in a large impact trial of mentoring and to explore whether a ‘trial-able’ shared-practice model could be identified and consistently delivered across organisations.
Following a rapid review of the literature, and two workshops with DPOs, the evaluators proposed a shared-practice model of mentoring. Elements of the shared practice model included using trained, paid adult mentors to deliver one-to-one, weekly mentoring, over a minimum of 12 weeks, to 10-17 year olds who were at risk of serious youth violence. Mentoring was voluntary for the young person. Nine DPOs delivered this model to 93 young people as part of a multi-site randomised controlled trial. 55% of children identified as being White; 26% as Black/Black British; 13% as Mixed; and 5% as Asian. 47 young people were randomised to a waiting-list control group and 46 young people were randomised into the intervention group. Young people in the control group were required to wait for 12 weeks before receiving mentoring. In the meantime, they received ‘business as usual’ activities, including music, football, art, and work experience. Young people in the intervention group received mentoring immediately.
Young people completed baseline and 12-week follow up surveys (the Strengths and Difficulties Questionnaire (SDQ) and the Youth Report of Socio-emotional Skills (YRSS)). Programme administrative data, a survey of all mentors, a mentoring quality survey from young people, and interviews with nine DPO managers, ten mentors, and nine mentees were also used. The evaluation team provided extensive support to DPOs to support their engagement in the evaluation. Recruitment of DPOs began in September 2021 and the trial ended with the completion of data collection in November 2022.
Key conclusions
| The MST arrangements were feasible and acceptable to the DPOs and young people. DPOs were successfully recruited, and they remained committed to the trial. All nine DPOs that started the feasibility study completed the evaluation and attended the majority of scheduled support meetings during the trial period. Good engagement was ensured by intensive support from the evaluators. |
| Mentee recruitment targets were largely met, with seven out of nine DPOs recruiting 10 or more young people. It was a challenge for some DPOs to meet recruitment targets, and the recruitment timeline was extended. |
| The shared practice model was deemed to be acceptable by 79% of mentors, and 89% of DPOs delivered the model with high to medium fidelity (67% with high fidelity). Data collection was also good. Eighty-three per cent of young people in the intervention group completed the follow-up survey, as did 87% of those in the control group. |
| Mentoring attendance was good. Seventy-six per cent of young people in the intervention group received eight or more mentoring sessions. Trial procedures were also largely followed. Only one young person in the control group reported receiving mentoring during their waiting period. |
| The evaluators recommend that the project moves forward to a larger trial. All eight pre-set progression criteria were met, and only minor adjustments are proposed to trial design and procedures. |
What will YEF do next?
YEF is proceeding with further evaluation, funding a larger pilot Multi-Site Trial.