Scottish Mentoring and Leadership Programme: impact and value for money evaluation

Appendix D - Value for Money estimates using Approaches C and D

For the purposes of illustrating the methodology for quantification and monetisation in Chapter 4, Approach B2 has been used. However, to reflect that estimates of monetised benefits vary depending on the analytical approach chosen from the impact estimates, the findings in the Social Cost Benefit Analysis (SCBA) are presented as ranges reflecting the differences when using Approaches B2, C or D. This appendix shares the full calculations for the Value for Money using the estimates from Approaches C and D which can be compared to the monetary estimates in the report using Approach B2 and are used in the ranges presented in the SCBA.

Educational attainment - using Approach C

Table D.3 in the supplementary tables^[42] sets out the calculations for size of effect of MCR Pathways programme on attainment using Approach C. The estimated effect is calculated as the difference of point estimates for target group and control groups which is used to estimate a comparison group. As the target group and control group have different sample sizes, the control group averages for each grade are applied to the sample size of the target group to estimate a comparable number of Level-5 qualifications achieved by grade. The difference between the control group totals (accounting for sample size) and target group totals provides the impact of MCR on educational attainment at Level-5.

This approach of estimating educational attainment impacts suggests that, based on Approach C, MCR Pathways programme resulted in the average pupil gaining 0.09 A grades, 0.11 B grades and 0.10 C grades compared to the counterfactual where this programme was not available, all else being equal. This equates to 311 A grades, 380 B grades and 345 C grades across the target group.

Table D.4 in the supplementary tables shows the monetised estimates of benefits from improving educational attainment according to Approach C. To estimate the total benefits in monetary terms the monetary estimates for the value of a grade are applied to the difference between target group and the comparison group (accounting for sample size). Using the above approach, we estimate that the impact on educational attainment equates to £42.6m of increased lifetime earnings in total. Although the benefits are quantified over the lifetimes of pupils (and therefore for decades into the future) the value is specific to MCR Pathways support delivered between 2014 and 2021.

Educational attainment - using Approach D

Table D.5 in the supplementary tables sets out the calculations for size of effect of MCR Pathways programme on attainment using Approach D. The estimated effect is calculated as the difference of point estimates for target group and control groups which is used to estimate a comparison group. As the target group and control group have different sample sizes, the control group averages for each grade are applied to the sample size of the target group to estimate a comparable number of Level-5 qualifications achieved by grade. The difference between the control group totals (accounting for sample size) and target group totals provides the impact of MCR on educational attainment at Level-5.

This approach of estimating educational attainment impacts suggests that, based on Approach D, MCR Pathways programme resulted in the average pupil gaining 0.09 A grades, 0.10 B grades and 0.09 C grades compared to the counterfactual where this programme was not available, all else being equal. This equates to 311 A grades, 345 B grades and 311 C grades across the target group.

Table D.6 in the supplementary tables shows the monetised estimates of benefits from improving educational attainment according to Approach D. To estimate the total benefits in monetary terms the monetary estimates for the value of a grade are applied to the difference between target group and the comparison group (accounting for sample size). Using the above approach, we estimate that the impact on educational attainment equates to £40.3m of increased lifetime earnings in total. Although the benefits are quantified over the lifetimes of pupils (and therefore for decades into the future) the value is specific to MCR Pathways support delivered between 2014 and 2021.

Unemployment

The impact of MCR Pathways on the number of people not in work or study has been quantified in Chapter 3. The findings show that 12% of the MCR Pathways programme cohort were unemployed 9 months after leaving school. The impact estimates for Approaches B2, C and D all show an effect size of 5 percentage points so we would expect 17% of the target group to have been unemployed in the absence of MCR Pathways. Applying the monetary estimates as shown in Chapter 4 this equates to £0.95m of benefits from MCR Pathways from reducing unemployment.

Exclusions - using Approach C

Table D.7 in the supplementary tables shows the size of effect of MCR Pathways programme on the number of exclusions by exclusion length using Approach C. The impact of MCR Pathways on reducing the number of exclusions and their severity is quantified in Chapter 3. The findings show that 8% of the MCR target group were excluded between 1-5 days in total, 5% for 6-10 days and 7% for more than 10 days equating to 276 pupils, 173 pupils and 242 pupils, respectively. The percentages of pupils excluded in the control group was higher than the target group with 10% of the control group excluded for between 1-5 days in total, 7% for between 6-10 days and 10% for more than 10 days. The estimated effect sizes are calculated as the difference of point estimates for target group and control groups. As the target group and control group have different sample sizes, the control group averages for each grade are applied to the sample size of the target group to estimate a comparable estimate of the number of exclusions for the control group across the different lengths of exclusions which equals 345 pupils excluded for 1-5 days, 242 pupils excluded for 6-10 days and 345 pupils excluded for more than 10 days. The difference between the control group totals (accounting for sample size) and target group total which equates to 69 pupils excluded for 1-5 days, 69 pupils excluded for 6-10 days and 104 pupils excludes for more than days which represents the additional number of exclusions we would expect in the absence of MCR Pathways.

Table D.8 in the supplementary tables shows the monetised estimates of benefits from reducing the number of exclusions and exclusion length according to Approach C. Applying the estimated costs of exclusion to schools to the difference between the difference between the target group total and control group totals (the number of avoided exclusions) results in a saving of £0.21m.

Exclusions - using Approach D

Table D.9 in the supplementary tables shows the size of effect of MCR Pathways programme on the number of exclusions by exclusion length using Approach D. The impact of MCR Pathways on reducing the number of exclusions and their severity is quantified in Chapter 3. The findings show that 8% of the MCR target group were excluded between 1-5 days in total, 5% for 6-10 days and 7% for more than 10 days equating to 276 pupils, 173 pupils and 242 pupils, respectively. The percentages of pupils excluded in the control group was higher than the target group with 9% of the control group excluded for between 1-5 days in total, 6% for between 6-10 days and 10% for more than 10 days. The estimated effect sizes are calculated as the difference of point estimates for target group and control groups. As the target group and control group have different sample sizes (the target group has a sample size of 3,452 compared to 2,590 for the control group) the control group averages for each grade are applied to the sample size of the target group to estimate a comparable estimate of the number of exclusions for the control group across the different lengths of exclusions which equals 311 pupils excluded for 1-5 days, 207 pupils excluded for 6-10 days and 345 pupils excluded for more than 10 days. The difference between the control group totals (accounting for sample size) and target group total which equates to 35 pupils excluded for 1-5 days, 35 pupils excluded for 6-10 days and 104 pupils excludes for more than days which represents the additional number of exclusions we would expect in the absence of MCR Pathways.

Table D.10 in the supplementary tables shows the monetised estimates of benefits from reducing the number of exclusions and exclusion length according to Approach D. Applying the estimated costs of exclusion to schools to the difference between the difference between the target group total and control group totals (the number of avoided exclusions) results in a saving of £0.19m.