Scottish Mentoring and Leadership Programme: impact and value for money evaluation

2. Analytical approach

The assessment of the quantitative impact of the SMLP programme requires a comparison between participants and an equivalent group of non-participants (a comparison group) to establish what may have occurred in the absence of the programme, so that any difference in outcomes can be confidently attributed to the programme. This chapter summarises our approach to this. Further technical details are provided in Appendix A.

Eligible population and definition of participation

The quantitative assessment of impact considers the MCR Pathways element of SMLP for Group 1 (care experienced) pupils only. Columba 1400 pupils and MCR Pathways Group 2 (˜tough realities) are not included in the analysis.

The exclusion of Columba 1400 pupils and MCR Pathways Group 2 from this analysis is because the criteria for participating in these are flexible^[3] and not defined on the basis of readily observable pupil characteristics. Instead, school-coordinators and/or other school staff have discretion about who should be offered a place on the programme at the local-level. As such, it is not possible to identify suitable control groups that would enable robust analysis for these groups.

Therefore, for this analysis, the eligible population is defined as MCR Pathways Group 1 pupils only; namely, pupils who had a record of any degree of experience in care, e.g. with fostering or adoption, or care within their families such as an aunt or grandparent (known as kinship care), or with a family friend, or in residential care.

Participants in the programme were defined as care experienced pupils who had received ten or more mentoring sessions.

Outcomes of interest

Measures of the outcomes of the programme were gathered by linking records of participation in the programme with pupil-level data held by the Scottish Government on attendance rates, attainment, staying-on rates, and positive post-school destinations.

There were several outcomes of interest:

• Attendance and exclusions. We report attendance in S4, S5, and S6 separately, comparing treatment and control groups. As data is collected biennially, the sample size is smaller than for other outcomes of interest.
• Exclusions. Exclusions are a low-frequency outcome and to ensure sufficient variance across individuals to support statistical analysis, we measure total exclusions throughout secondary schooling. These measures will include data on exclusions among care experienced pupils in years prior to intensive engagement in the MCR Pathways programme, though as this is also the case for the comparator group, this will not distort or bias estimates of impact. Data on exclusions is also collected biennially.
• Staying-on rates: Two types of measures of staying-on were created. The first was based on evidence of last year of schooling (whether S4, S5 or S6) based on the Pupil Census. However, the Pupil Census is undertaken early in the academic year, so may not reflect pupils who leave midway through an academic year. The other measure created was if pupils had achieved qualifications in two or more exam diets. This is indicative of staying-on until examinations at the end of each academic year.
• Attainment. Qualifications were summarised across different years per pupil. Several summary measures were calculated:
  • A qualification in English at SCQF Level-5 or above
  • A qualification in Maths at SCQF Level-5 or above
  • A minimum of 3 SCQF Level-5 qualifications
  • A minimum of 5 SCQF Level-5 qualifications
  • A minimum of 3 SCQF Level-6 qualifications
  • A minimum of 5 SCQF Level-6 qualifications
  • A minimum of 1 SCQF Level-7 qualification
• Positive post-school destinations at 3 months and 9 months post-leaving school as provided within the Destination of Leavers from Scottish Schools datasets.

Data sources and notes on data processing

The following Scottish Government data sources were used:

• Pupil Census data (16 annual waves from 2008/2009 to 2023/2024^[4]).
• Attendance data (9 biennial waves^[5] from 2008/2009 to 2022/2023).
• Leaver destinations (16 annual waves from 2008/2009 to 2023/2024).
• Care experience (1 file covering the school academic years 2008/2009 to 2023/2024).
• Pupil additional support needs and reasons for support (16 annual waves from 2008/2009 to 2023/2024 onwards).
• Qualifications (16 annual waves from 2008/2009 to 2023/2024; qualifications were summed across waves to produce measures of attainment).
• Exclusions data (9 biennial^[6] waves from 2008/2009 to 2022/2023).

Where the format varied between waves, data was harmonised to allow comparison over time.

This data was combined with files from MCR. These included: whether pupils had participated in MCR Pathways; whether these pupils had been mentored (had received ten sessions or more) or non-mentored; and MCR-collected data on the type of care experience of the participants. The MCR data covered pupils from 2016/17 to 2022/23.

Data on schools was collated from Administrative Data Research (ADR) Scotland. This covered: denomination of school, rurality, area deprivation profile of pupils, percentage of pupils from minority ethnic backgrounds, percentage of pupils with additional support needs and additional language needs. The school-level data covered each year from 2008/2009 onwards. The Scottish Government also separately provided measures of pupil attainment at school-level.

Discrepancy on care experience data

The Children Looked After Survey (CLAS) collates annual returns from local authorities and provides details of children in local authority care. It should be noted that there was a discrepancy between the MCR data and the CLAS data on whether pupils had experienced care. Not all MCR participants flagged as care experienced appeared in the CLAS dataset. Further details are provided in Appendix A.

To ensure consistency between how the control group and the participant group were defined in the analysis, the source used for defining pupils with care experience was the Children Looked After Survey (CLAS) rather than the MCR data on care experience.

Summary of the different analytical approaches

Five different analytical approaches were used to assess the impact of the programme. All five approaches compare a treatment group to a control group (formed of either pupils - Approaches A, B1 and B2 - or schools - Approaches C and D). Approaches A, B1 and B2 provide point estimates of impact, while Approaches C and D estimate marginal effects which are then converted into point estimates. The different approaches vary in their robustness and statistical power. These are described below. In broad terms, they can be divided into pupil-level and school-level analyses:

• Pupil-level analysis: Approaches A, B1, and B2 (see below) infer the impacts of MCR Pathways by comparing outcomes for care experienced pupils in participating schools with care experienced pupils in non-participating schools. However, comparisons between participating and non-participating pupils could be biased if there are systematic differences between the two groups of pupils (e.g. in terms of motivation or ability) or between schools (e.g. in terms of their ability to meet the needs of care experienced pupils). Steps were therefore taken incrementally to control for observable differences between the two groups of pupils in these approaches (with Approach B2 considered the most robust). However, it was not possible to apply approaches that are robust to unobservable differences between pupils or schools using a pupil-level approach (such as difference-in-differences models). These approaches can only be applied where outcomes can be observed both before and after the intervention. In this case, at the pupil-level, the outcomes of interest can only be observed post-intervention (e.g. staying-on rates or attainment). School level analysis partially resolves this issue by allowing for differences in outcomes before intervention.
• School-level analysis: Longitudinal data on the outcomes of interest can be constructed at the school-level by examining aggregate outcomes for cohorts of care experienced pupils both before and after the introduction of MCR. This involves comparing the proportions and averages of outcomes for the care experienced population, e.g. the proportion going on to Higher Education or the proportion achieving particular qualifications in different schools. Thus, Approaches C and D infer impacts by comparing cohorts of care experienced pupils in participating and non-participating schools, while controlling for past performance through regression modelling. They are therefore robust to unobserved differences between schools that do not change over time. While these approaches are more robust (in terms of causal inference), the sample sizes of participating schools are smaller than the sample sizes of participating pupils, weakening the statistical power of any estimates.

There is, therefore, a trade-off in terms of robustness and statistical power: there is a risk that Approaches A and B overstate effect sizes and a risk that Approaches C and D fail to detect effects due to small sample size.

In practice, the findings from Approach B2 (which controls for differences in pupil and school characteristics) and the school-level analyses (C and D) were broadly comparable - indicating that findings for B2 are not significantly biased by the presence of unobserved factors. As such, the main analysis of effects in this report uses Approach B2 to define the statistical significance of effects and Approaches B2, C and D to estimate the range of effect sizes.

Approach A: Participants versus a matched group of non-participants.

This approach estimates the Average Treatment Effect (ATE) by comparing the specific outcomes of individuals who have participated in the MCR Pathways programme with a matched control sample of care experienced pupils from non-participating schools that share similar measurable characteristics, namely: gender, ethnicity, disability, English language proficiency, national identity, refugee status and additional support needs. The selection of the control group uses propensity score matching techniques. (Full details are provided in Appendix A).

The strength of this approach lies in its ability to focus on the impact experienced by programme participants (i.e. those who took part, rather than simply those eligible). However, all key factors that drive participation need to be measurable for results to be robust. In practice, it is likely that there are significant unobservable factors that influence participation in the programme, making it difficult for the matching to produce a control group that is completely comparable to participants. As such, this method is at risk of producing findings that overstate the impact of the programme. In other words, among all those who are eligible to take part in the programme, those who are most likely to have positive post-school outcomes are also the most likely to take part in the programme, and we can only partially control for this by matching on observable characteristics.

Approach B1: Care experienced pupils in participating schools vs care experienced pupils in all non-participating schools

The second approach expands the definition of the ˜treatment group to include all care experienced pupils in participating schools rather than just programme participants and compares them to all care experienced pupils in non-participating schools (i.e. an Intention-to-Treat estimator).

This removes possible biases driven by differences between pupils that do and do not take up the programme in participating schools. However, it is still based on individual-level comparisons and still carries some of the weaknesses associated with Approach A. For instance, if schools participating in MCR also provide a higher level of support towards care experienced pupils success more generally compared to non-participating schools, this approach will also overestimate the programmes effects.

Approach B2: Care experienced pupils in participating schools vs care experienced pupils in matched non-participating schools

Approach B2 refines the Approach B1 so that the control group of care experienced pupils is selected only from matched non-participating schools. This involved matching MCR schools with non-MCR schools on observable characteristics at the school-level using propensity score matching. These characteristics were: school roll size, past leaver outcomes, past qualification outcomes, past exclusion rates and local employment and wage rates at the Local Authority level. Overall, this approach is more robust than B1, as it accounts for school-related factors that might influence effect sizes. It has less statistical power than B1, however, as the sample size of the control group is smaller.

Approach C: Comparison between participating and non-participating schools

The fourth approach involves reconfiguring the analysis from the pupil-level to the school-level. This involved analysis of aggregate outcomes in terms of pupil leaver destinations (proportion of care experienced pupils leaving for each destination), cohort attainment outcomes (proportion of care experienced pupils attaining qualifications and at what level) and cohort exclusion outcomes. The use of regression analysis to control for past observable outcomes at the school-level means that this method is more robust and less likely to be impacted by possible distortions driven by unobserved differences between pupils and schools. Schools participating in MCR were matched with control schools, again using Propensity score matching.

While offering a more robust framework for the analysis than the pupil-level analysis, this approach is likely to have less statistical power for a variety of reasons: small sample sizes; potential for greater variance in the outcomes of interest; and dilution of effects from eligible pupils who do not take part in the programme. An effect size that is statistically significant in Approach B2 may not be statistically significant with a similar effect size in Approach C. This means there is a risk that effects of the programme are not detected with this approach.

Approach D: Staggered treatment design

The final approach was a staggered treatment design. This approach makes comparisons between schools that adopted MCR earlier versus those that joined later. The impacts of MCR participation should be visible amongst the cohort of schools joining earlier. This is potentially more robust than the previous approaches as comparisons are only made between schools that ultimately participate in the programme which are more likely to share unmeasurable characteristics that influenced participation in MCR and/or the outcomes of interest.

As with Approach C, while this approach is likely to be more robust than the pupil-level analysis (Approaches A and B), it has considerably less statistical power and therefore risks not detecting effects.

Further details of the different approaches are provided in Appendix A.

Year groups included in the analysis

Table 2.3 shows details of the number of S4 pupils by year. The analysis is based on those in S4 between 2014/2015 and 2021/2022. This is because we have no records of MCR participants in earlier cohorts, while the latest data that is available for outcomes such as qualifications and destinations is 2023/24, meaning that the S4 in 2021/2022 cohort is the last cohort with full data.