Evaluation type
What does this project involve?
The Educate Mentoring Programme aims to improve children’s wellbeing, resilience, social relationships and confidence, and lead to a long term reduction in offending. Delivered by the Rugby Football League (RFL) across Leeds, Huddersfield, Warrington, St Helens, Hull, Wigan and Leigh, Educate Mentoring is a 12 week programme targeted at 11-14 year olds. Weekly mentoring sessions are offered, which focus on building core personal skills and improving children’s awareness of risky behaviours and positive choices. Sessions, delivered by RFL professional club foundation coaches, are typically provided in school, last two hours, and comprise a mentoring conversation followed by physical activity. ‘At risk’ young people are selected for Educate Mentoring by schools, who target children with poor behaviour and attendance, and an interest in sport.
Why did YEF fund this project?
As the YEF’s toolkit explains, both sports programmes and mentoring interventions are associated with a positive average impact on reducing serious youth violence and crime. However, there are considerable gaps in the evidence, particularly relating to robust evaluations conducted in an English or Welsh context.
YEF, therefore, funded a feasibility and pilot evaluation of Educate Mentoring. The feasibility study aimed to ascertain whether the programme could achieve its intended outputs for the intended target groups, explore the facilitators and barriers to delivery, detail how many mentoring sessions children received, and assess the quality, responsiveness, and reach of the programme. To explore these questions, the evaluators interviewed 17 pupils, and 14 project staff and partners. Project delivery monitoring data on 218 programme participants was also analysed, alongside an online satisfaction survey with 82 children.
The pilot study then aimed to assess the extent to which an efficacy randomised controlled trial of Educate Mentoring might be feasible, acquire detailed information that could inform the design of a future evaluation, and ascertain whether there was any preliminary evidence of promise. To explore these questions, the evaluator analysed demographic data and project delivery data relating to 111 pupils, administered surveys that included validated measures (the Strengths and Difficulties Questionnaire (SDQ) and Problem Behaviour Frequency Scale (PBFS)), and conducted interviews and focus groups with 8 project staff, 7 teachers, and 29 pupils. The pilot was delivered as a randomised controlled trial, with 10 schools and 111 pupils; schools recruited to the pilot were allocated to either receive the programme, or to a waitlist control group. 88% of the young people involved identified as White, 5% as Asian, 4% as Black, and 4% as Mixed ethnicity. The evaluation ran from November 2019 to June 2022.
Both the feasibility and pilot studies took place during the coronavirus pandemic, requiring both the delivery and evaluation teams to adapt to challenging circumstances.
Key conclusions
| The feasibility study found that 218 young people received the programme across five areas. Mentoring was generally delivered as intended; the aims, structure, themes and session content corresponded to the intended design. No significant barriers to delivery were identified, and the challenges posed by COVID-19 were effectively surmounted. |
| Two-thirds of the participants captured in feasibility study monitoring data (116/177) received 12 or more sessions, thus completing the programme. Young people reported that the programme was engaging and addressed their expectations and needs. The lived experience of coaches, the relational nature of the programme and the novelty of the non-school activities featured in the intervention were perceived to support engagement. |
| In the pilot study, RFL recruited 10 schools (out of a target of 14). They did not meet the target, in part, because of a requirement to not recruit schools that had previously been involved in the programme, while COVID-19-related challenges and short time frames also hampered recruitment. The project team was able to explain the trial to schools; school staff understood randomisation; and RFL, schools and parents found the design acceptable. |
| Schools recruited an average of 11.6 pupils per school to the project in the pilot (slightly under the target of 12). 100% of pupils provided data at baseline. 78% provided data 5 months after randomisation. Due to a smaller sample size than anticipated, the evaluator was unable to assess whether there was evidence of promise. |
| In the pilot study, as in the feasibility phase, RFL reported no significant adaptations to the delivery of the programme. The programme’s intended sessions were generally delivered, and in 4 out of 5 of the intervention schools, all pupils who commenced the programme were recorded as completing the sessions. |
What will YEF do next?
Given the proliferation of sports-based programmes that aim to divert at-risk young people from criminal activity, and the sizable gaps in the evidence base relating to these programmes, the evaluator recommends that YEF considers funding an efficacy randomised controlled trial of the Educate Mentoring programme. YEF is, therefore, exploring whether an impact evaluation is possible.