Scottish Mentoring and Leadership Programme: impact and value for money evaluation
Findings of the impact and value for money evaluation of the Scottish Mentoring and Leadership Programme (SMLP), looking at the MCR Pathways element of the programme for care experienced young people.
Executive summary
This evaluation report sets out the quantitative evidence of the impact of the MCR Pathways programme and the evidence of Value for Money. The evaluation demonstrates that the MCR Pathways programme has had a positive impact on pupil outcomes. Pupils participating in the programme are more likely to pursue Further Education and training, while showing a decreased likelihood of unemployment. Although some effects may weaken over time, the programme contributes to reducing unemployment and supporting pupils into positive post-school destinations in education and work. The programme significantly benefits care experienced pupils educational achievements and post-school prospects. Its effectiveness in lowering exclusion rates, enhancing staying-on rates, boosting academic performance, and improving post-school outcomes highlights its positive role as a supportive intervention for care experienced pupils. It helps to narrow the considerable gap between care experienced and non-care experienced pupils on such measures.
The evaluation estimates that for every £1 spent on MCR Pathways for care experienced pupils, between £2.30 and £2.80 were returned in societal and economic benefits depending on which approach from the impact estimates is used. These findings, alongside wider commentary on additional benefits that could not be monetised in this evaluation, provide strong evidence of value for money.
Background
The Scottish Mentoring and Leadership Programme (SMLP) is designed to support and improve the outcomes of care experienced young people, and those in ˜tough realities, such as those on the cusp of care, young carers, and those living in SIMD 1 or 2 areas.
SMLP comprises two key components: MCR Pathways and Columba 1400 Young Peoples Leadership Academies. MCR Pathways is a school-based mentoring programme, engaging S1 and S2 pupils in group work and matching pupils with local mentors who meet with them in school once a week from S3. Columba 1400 provides values-based leadership experiences for young people, delivered in three phases over S2 and S3.
This analysis only covers the MCR Pathways element of SMLP for care experienced pupils. It does not include the Columba 1400 element or the MCR Pathways element offered to non-care experienced pupils in ˜tough realities. As the inclusion criteria for these vary, it was not possible to identify suitable control groups to assess the impact with sufficient robustness.
The estimated impact of the MCR Pathways programme on outcomes was based on data that linked records of participation in the programme with pupil-level data held by the Scottish Government on attendance rates, attainment, staying-on rates, and post-school destinations. The analysis is based on those in S4 between 2014/2015 and 2021/2022.
Impact Evaluation findings
The impact evaluation was conducted using five analytical approaches, with the three most robust approaches used to estimate the impact of the programme on outcomes. Outcomes for participating pupils and schools were compared against outcomes for pupils and schools where there was no MCR Pathways participation, with the application of statistical matching methods ensuring comparisons were made between similar groups.
The quantitative impact evaluation of the MCR Pathways programme revealed significant positive impacts on various educational and post-school outcomes for care experienced pupils. The table below outlines the estimated range of impact the MCR Pathways programme has had on participating care experienced children and young people. This has been estimated using a selection of statistical tests, but these are reported in a single range here to demonstrate the range in magnitude of impacts that have been identified within the evaluation.
| Measurable outcome | Estimated range of effectiveness of the programme |
|---|---|
| Ever excluded from secondary school | 7pp to 9pp reduction |
| Staying on to S5 | 8pp to 13pp increase |
| Achieving 3+ Nat-5 qualifications | 6pp to 7pp increase |
| Positive post-school destination 9 months after leaving school | 5pp to 7pp increase |
The evaluation did not find robust evidence that MCR has improved attendance rates overall, although it does suggest that the programme has potentially led to an improvement in S5 attendance.
The analysis covered the impact of the MCR Pathways programme for care experienced pupils only. To give context to the estimated effect sizes, it is useful to compare these against the difference in outcomes between all non-care experienced and all care experienced pupils across Scotland. This is summarised in the table overleaf.
| Measurable outcome |
Share of non-care experienced pupils who¦ |
Share of care experienced pupils who¦ |
Difference |
|---|---|---|---|
| ¦were ever excluded from secondary school | 6% | 29% | -23pp |
| ¦stayed on to S5 | 89% | 68% | +21pp |
| ¦achieved 3+ Nat-5 qualifications | 75% | 27% | +48pp |
| ¦were in a positive post-school destination 9 months after leaving school | 90% | 70% | +20pp |
Sources: Pupil Census, all pupils in secondary school in S4 between 2014/2015 to 2021/2022; care experience defined by Children Looked After Survey (CLAS); Scottish Government collated data on attainment based on official exam results certified by the Scottish Qualification Authority (SQA); Scottish Government exclusions from school data.
The increase in staying-on rates of 8 to 13 percentage points as a result of MCR Pathways is equivalent to between a third and almost two thirds of the gap in staying-on rates between non-care experienced and care experienced pupils. This is the outcome where MCR Pathways has narrowed the gap the most. The impact of MCR Pathways on fewer exclusions equates to around one third of the gap in the exclusion rates between care experienced and non-care experienced pupils, while the increase in positive post-school destinations as a result of the programme is the equivalent of around a quarter to a third of the gap in this outcome.
Improvements in attaining three or more Level-5 qualifications of 6 to 7 percentage points represent less than a fifth of the gap between care experienced and non-care experienced pupils, narrowing the gap to a lesser extent here than for the other outcomes measured.
Value for Money findings
The Value for Money assessment estimates the costs of delivering MCR Pathways against the monetised benefits, which included the benefits of improved educational attainment, reduced unemployment, and avoided school exclusions. This assessment also included the wellbeing gains of volunteers, acknowledging the role volunteers play in the MCR Pathways programme in supporting the pupils they mentor on a one-to-one basis. Many of these benefits, such as the benefits of improved educational attainment to lifetime earnings, will be realised a considerable time after the evaluation period, as they are calculated over the pupils lifetime.
These estimates underpin a Social Cost Benefit Analysis, which assesses the impact of the programme on social welfare, with all relevant costs and benefits valued in monetary terms (where proportionate and possible).
Two metrics were assessed - the ratio of benefits to costs, the Benefit Cost Ratio (BCR), and the value of benefits minus the costs, the Net Present Value (NPV). All BCR and NPV estimates are consistent with HM Treasury Green Book Guidance on assessing Value for Money.
Compared to an estimated cost of £18.8 million for delivering this element of the MCR Pathways programme, the benefits outweigh the cost with a Benefit Cost Ratio (BCR) of between 2.3:1 and 2.8:1, and a Net Present Value (NPV) of between £24.3 million and £34.6 million. This means that for every £1 spent on MCR Pathways for care experienced pupils, between £2.30 and £2.80 of benefits to society are created, with the benefits outweighing the costs by at least £24.3 million and up to £34.6 million. This suggests that investment in the programme represents value for money when considering the societal and economic benefits.
There is potential additional value from the MCR Pathways programme not included in these calculations, including from possible reductions in homelessness, crime, and NEET outcomes, as well as improving attendance and educational attainment at higher SCQF Levels. These benefits are not included in the calculations of the BCR and NPV within the Social Cost Benefit Analysis. This is either because these impacts have not been quantified in this evaluation (either due to data availability or the need to prioritise evaluating other outcomes), their inclusion could result in at least partial double counting with some of the benefits already accounted for in the Social Cost Benefit Analysis, or the existing literature is not developed enough to provide robust monetary estimates. Given the potential impact of these wider benefits, however, the figures in this report are more likely to under- than over-estimate the overall value of the programme.