Scottish Mentoring and Leadership Programme: impact and value for money evaluation

5. Concluding summary

This report has provided the results of the quantitative impact evaluation of the MCR Pathways element of the SMLP programme. Columba 1400 pupils and MCR Pathways Group 2 (˜tough realities) pupils have been excluded from this analysis because their eligibility criteria for participating are flexible and, as such, it is not possible to identify suitable control groups that would enable robust analysis.

To ensure robust analysis and confidence in the results, the impact evaluation was conducted using five analytical approaches. The impacts have been reported using the three most robust Approaches B2, C and D, that all estimate the Intention-to-Treat effect.

The findings have revealed significant positive impacts on almost all of the educational and post-school outcomes considered for care experienced pupils. The programme has had a positive impact on reducing exclusion rates, increasing staying-on rates, improving attainment, and improving post-school destinations.

• Exclusions. The MCR Pathways programme has had positive effects in terms of reducing school exclusions amongst the target group. The estimated share of care experienced pupils ever being excluded fell by seven to nine percentage points, depending on the analytical approach taken.
• Staying-on Rates. The programme has had a positive impact on increasing rates of staying-on past the end of compulsory education. All analytical approaches indicate a significant positive effect on staying on to S5, with estimated effect sizes ranging from eight to thirteen percentage points. The programme also positively impacts the rate of pupils staying-on to S6 and increases the proportion of pupils achieving qualifications in more than one annual exam diet (another indicator of staying on longer in school).
• Attainment. The programme shows a significant positive effect on attainment, with the magnitude varying across different qualification levels. The impact is greater for less advanced courses. While the effect sizes appear small when presented as a percentage point change, the impact of the programme should not be underestimated. For example, given that among all care experienced pupils, just 10% achieve 3 or more Level-6 qualifications, the estimated effect of the programme of a three to four percentage point increase is considerable. The estimated effect size range is:
  • Six percentage points for achieving 1 or more Level-5 qualifications in English.
  • Three to four percentage points for 1 or more Level-5 qualifications in Maths.
  • Six to seven percentage points for achieving 3 or more Level-5 qualifications.
  • Three to five percentage points for achieving 5 or more Level-5 qualifications.
  • Three to four percentage points for achieving 3 or more Level-6 qualifications.
• Post-School Destinations. The programme has had a positive effect on pupils' post-school destinations, particularly in increasing participation in Further Education and training and reducing unemployment. The programme's impact on reducing unemployment is maintained over time, with an estimated effect size of a five percentage point reduction nine months after leaving school. The effect of the programme on increasing the proportion of care experienced pupils being in any positive outcomes post-school also remains considerable nine months after leaving, although the size of the impact is smaller than at three months after leaving.

The evaluation did not find that MCR Pathways has had a clear impact on attendance rates overall, although it does suggest that it has led to an improvement in S5 attendance.

The analysis covered the impact of the MCR Pathways programme for care experienced pupils only. To give context to the estimated effect sizes, it is useful to compare these against the difference in outcomes between all non-care experienced and all care experienced pupils across Scotland.

• The increase in staying-on rates of 8 to 13 percentage points because of MCR Pathways is equivalent to between a third and almost two thirds of the gap in staying-on rates between non-care experienced and care experienced pupils. This is the outcome where MCR Pathways has narrowed the gap the most.
• The impact of MCR Pathways on fewer exclusions equates to around one third of the gap in the exclusion rates between care experienced and non-care experienced pupils, while the increase in positive post-school destinations because of the programme is the equivalent of around a quarter to a third of the gap in this outcome.
• Improvements in attaining three or more Level-5 qualifications of 6 to 7 percentage points represent less than a fifth of the gap between care experienced and non-care experienced pupils, narrowing the gap to a lesser extent here than for the other outcomes measured.

The robustness of the statistical analysis was ensured through a range of methods applied to provide confidence in the estimated effects. This improves our confidence in the stated effects. The five different analytical approaches used for the impact analysis were:

• Approach A: Care experienced participants vs matched care experienced non-participants.
• Approach B1: All care experienced pupils in participating schools (i.e. including those who do not take part in the programme) vs care experienced pupils in all non-participating schools.
• Approach B2: Care experienced pupils in participating schools vs care experienced pupils in matched non-participating schools. This involves matching schools to ensure the control group is from schools that are similar to those that have adopted MCR Pathways.
• Approach C: Comparison between participating and non-participating schools, reconfiguring the analysis from individual level to school/cohort level, controlling for past outcomes at the school level.
• Approach D: A ˜staggered treatment design that compares outcomes at the school level, similar to Approach C, but comparing schools that adopted MCR earlier with those that adopted MCR later.

Approaches B2, C, and D were the most robust Approaches, with B2 offering the best balance between statistical power and robustness. Application of matching methods ensured comparisons were being made between similar pupils. Approach B2 took into account school level confounding factors. Matching and regression analysis of school level outcomes through Approach C then controls explicitly for confounding factors at the school level and past school performance, with the final Approach D providing comparisons amongst the most similar schools that all adopted MCR at some point. The analysis of the impact of the programme on the different outcomes was therefore based on the estimated effect of these three approaches, with Approach B2 used to determine if the effects were significant.

The Value for Money assessment built on the findings from the impact evaluation to estimate a Benefit Cost Ratio for the programme, developed in line with HM Treasury Green Book Guidance. Compared to an estimated cost of £18.8 million, the benefits from the MCR Pathways programme outweigh the cost, with a Benefit Cost Ratio (BCR) of between 2.3:1 and 2.8:1, and a Net Present Value (NPV) of between £24.3 million and £34.6 million. This means that for every £1 spent on MCR Pathways, between £2.30 and £2.80 of benefits to society are created, with the benefits outweighing the costs by between £24.3 million and £34.6 million. This suggests that investment in the programme represents value for money when considering the societal and economic benefits. There is also a range of additional benefits that could not be monetised which further suggest that this programme represents good value for money.

In conclusion, the evaluation has found that the MCR Pathways programme has had a considerable positive impact on care experienced pupils' educational and post-school outcomes. The programme's success in reducing exclusions, increasing staying-on rates, improving attainment, and enhancing post-school destinations underscores its value as an effective intervention for supporting young people with care experience, as suggested through the range of analytical approaches undertaken and the Value for Money assessment.