Scottish Mentoring and Leadership Programme: evaluation - final report

Chapter 7: Conclusions and suggestions for improvement

This chapter summarises the key findings from the evaluation in relation to the evaluation questions and offers suggestions for improvement.

Perhaps the most important finding from the evaluation is the hugely beneficial – and sometimes transformative – impact that relationship-based support can have for vulnerable young people.

To what extent has the programme supported the capabilities of young people?

There is quantitative evidence that MCR Pathways had a positive impact on a range of ‘hard’ outomes (like attainment and positive post-school destinations) for care experienced pupils. There is also qualitative evidence that both interventions – MCR Pathways and the Columba 1400 YPLA – have had a positive impact on the capabilities of the young people who have taken part in them (both care experienced and non-care experienced pupils). Increased confidence was identified as one of the main impacts of both interventions, and most of the capabilities in the programme Theory of Change (Appendix 1) were positively impacted. These impacts were mostly sustained among those re-interviewed two years later, in wave two.

There are limitations on robustly measuring the long-term impact of SMLP on young people within the scope and remit of this evaluation. However, evidence of the potential longevity of impacts was provided by the wave two findings on both MCR Pathways and Columba 1400, which confirmed how several impacts of the intervention on young people’s capabilities had been sustained two years on.

MCR Pathways qualitative impact findings

Increased confidence was a dominant theme when discussing the key impacts of MCR Pathways on young people, particularly in relation to increased social confidence and more belief in their own abilities. This increase in confidence was attributed, in part, to encouragement from mentors and MCR Co-ordinators. These individuals were described as ‘patient’, ‘non-judgemental’ and ‘reliable’ which allowed them to build strong relationships with pupils. These impacts were clearly sustained over time.

Other improvements, which different MCR pupils saw as having the most impact on them personally, were:

• Feeling less anxious
• Improved attendance at school
• Feeling calmer
• Feeling less angry
• Feeling more confident about doing well at a job in future.

MCR Pathways quantitative impact findings

The quantitative impact analysis (looking at outcomes for care experienced pupils only) revealed significant positive impacts on almost all of the educational and post-school outcomes considered. It found:

• Exclusions: positive effects in terms of reducing school exclusions amongst the target group. The estimated share of care experienced pupils ever being excluded fell by seven to nine percentage points, depending on the analytical approach taken.
• Staying-on Rates: a positive impact on increasing rates of staying-on past the end of compulsory education. All analytical approaches indicate a significant positive effect on staying on to S5, with estimated effect sizes ranging from eight to thirteen percentage points. The intervention also positively impacts the rate of pupils staying-on to S6.
• Attainment: a significant positive effect on attainment, with the magnitude varying across different qualification levels. The impact is greater for less advanced courses. The estimated effect size for different numbers/subjects of Level-5 qualifications ranges from three to four percentage points (for 1 or more Level-5 qualifications in Maths) to six percentage points (for achieving 1 or more Level-5 qualifications in English).
• Post-School Destinations: a positive effect on pupils' post-school destinations, particularly in increasing participation in Further Education and training and reducing unemployment. The intervention’s impact on reducing unemployment is maintained over time, with an estimated effect size of a five percentage point reduction nine months after leaving school. The effect of the intervention on increasing the proportion of care experienced pupils being in any positive outcomes post-school also remains considerable nine months after leaving, although the size of the impact is smaller than at three months after leaving.
• Attendance: The evaluation did not find robust evidence that MCR Pathways has improved attendance rates overall.

Value for Money assessment of MCR Pathways for care experienced pupils

The Value for Money assessment built on the findings from the impact evaluation to estimate a Benefit Cost Ratio for the intervention, developed in line with HM Treasury Green Book Guidance. Compared to an estimated cost of £18.8 million, the benefits from MCR Pathways outweigh the cost, with a Benefit Cost Ratio (BCR) of between 2.3:1 and 2.8:1, and a Net Present Value (NPV) of between £24.3 million and £34.6 million. This means that for every £1 spent on MCR Pathways, between £2.30 and £2.80 of benefits to society are created, with the benefits outweighing the costs by between £24.3 million and £34.6 million. This suggests that investment in the intervention represents value for money when considering the societal and economic benefits. There is also a range of additional potential benefits that could not be monetised which further suggest that this intervention represents good value for money. These include an increased number of SQA Level 4 qualifications (for which there are no monetised estimates), a reduction in crime and a reduction in homelessness.

Columba 1400 Young People’s Leadership Academy impacts

Participants were very positive about the Columba 1400 YPLA overall, describing the impact it had as ‘powerful’, ‘affirming’, ‘life-changing’ and ‘transformational’.

Increased social confidence and communication skills were again common themes when discussing the impacts of the residential model YPLA. Pupils attributed this to the amount of group work that they experienced, which encouraged them to speak to their peers and helped them feel more comfortable in group settings. Staff also mentioned that some pupils had been chosen to take part because they were struggling with confidence, and they had observed significant changes in those pupils. These impacts had mostly been sustained and in some cases had strengthened over time.

Pupils also highlighted how the YPLA with a residential element had improved their self-esteem and sense of belief in their abilities. They described having more confidence in themselves to contribute at school and to socialise or to try new things outside of school. Parents/carers and staff also observed more self-assuredness in the young people who, before taking part in Columba 1400, had lacked this. In general, these impacts had been sustained two years later.

In wave one, it was recognised that the residential element was a barrier for those young people in challenging circumstances for whom going away would be impractical or inappropriate. Columba 1400 staff noted that the place-based model - a new approach which involves activities within the local community and does not include a residential element - had been developed to engage with these young people. Evidence gathered from the place-based model case studies in wave two suggested that this alternative approach to delivering the YPLA had been successful in reaching different groups of young people (i.e. those for whom the residential would not be accessible). As with the residential model, increased social confidence was one of the main impacts of the place-based model – and this was felt to have underpinned many of the other capabilities that were impacted, such as belief in their abilities and enhanced social capital.

What worked well

School staff and wider stakeholders generally agreed that both interventions were reaching the young people who would benefit the most and were successfully engaging with them. The positive relationships between young people and the MCR School Co-ordinators/Columba 1400 delivery staff was seen as key to this successful engagement. In addition, the importance of school staff having the flexibility to decide which pupils take part – and trusting schools to make the right decisions – was emphasised by stakeholders.

MCR was generally felt to have a positive working relationship with other initiatives supporting young people, specifically Developing the Young Workforce, Skills Development Scotland, and the Promise.

Columba 1400 staff also mentioned engagement with other organisations supporting young people, including involvement with Developing the Young Workforce’s ‘World of Work’ annual event and conversations with Skills Development Scotland about developing a ‘leadership passport’.

How is the programme delivery changing and improving over time?

There were examples from both interventions of improvements to delivery based on MCR and Columba 1400 staff reflecting on how things could be better or based on feedback from schools, pupils and delivery staff.

For MCR, this included being more flexible with the model – allowing Co-ordinators to cover more than one school and dropping the S1 and S2 group work to make this feasible. This reduced the costs for the school/local authority and made the intervention more sustainable. They are also expanding the scheme into Further Education colleges to allow existing mentoring relationships to continue post-school and to thereby help sustain positive destinations.

For Columba 1400, the biggest change has been the introduction of the place-based model. This enables a wider range of young people to take part and to benefit from the YPLA.

Has the programme been implemented as intended?

MCR Pathways was generally felt to have been implemented as intended. There had been some local changes to school Co-ordinator employment terms and the delivery of group work. These changes were intended to improve delivery in terms of reducing staff costs (and therefore increasing the sustainability of the scheme), staff retention and intervention effectiveness for young people.

It was also generally agreed that the Columba 1400 YPLA residential and place-based models have been implemented as intended. A number of key considerations were highlighted for ensuring successful delivery, including:

Setting clear expectations with key adults supporting the delivery of the YPLA, whether in school or other organisations. This was felt to be critical to ensuring that the impacts on young people are sustained over the longer term.

Making the most of Phase Three, as this was highlighted as a key stage for embedding and strengthening the ‘being’ and ‘doing’ capabilities. Maintaining a flexible approach to Phase Three was also felt to be important, as different groups have different capacities for undertaking project work.

Giving thought to what age and stage is best for the particular cohort being engaged. For schools, S2 was generally considered to be the best time for taking young people out of school for a week and providing opportunities for them to take on leadership roles in school in later years. Organisations that took part in the place-based model engaged a wider range of age groups; in particular, engaging young people at the ages of 14-16 was felt to be beneficial for helping them transition from school to positive destinations such as further education or work.

In relation to the SMLP programme as whole, the speed and scale of rollout was less than originally intended. The vision of the two interventions complementing each other within individual schools has also not materialised. Stakeholders indicated that the main reason for the more limited rollout of the programme was the much less favourable fiscal climate which had developed since the start of the programme. In 2022/23 there was also a structural change, with The Hunter Foundation funding only the Columba 1400 element of the programme. Scottish Ministers however remain committed to delivering the overall outcomes of the SMLP programme and will keep both elements under review, directly funding MCR Pathways and providing some funding to Columba 1400 for the development of Fairer Futures Partnerships and match funding in partnership with The Hunter Foundation for their Head Teachers’ Leadership Academies.

Have mentoring and leadership been embedded across the school system?

MCR Pathways, by 2021/2, had been embedded in all 30 Glasgow schools (mainly from 2014/15 onwards, after the first school came on board in 2007/8) and in 74 schools across the rest of Scotland, with these schools coming on board from 2017/18 onwards. There was an overall sense among participants that it is important for organisations like MCR to take the lead on embedding mentoring in schools, since school staff have other priorities and do not have the capacity to do this kind of work.

There was, however, limited evidence that mentoring has been embedded more widely across the local authority areas we visited outside of Glasgow. One local authority stakeholder, for example, wanted to see more data on the impact of the intervention before deciding whether to roll it out to other schools in their area. This is linked to the more limited roll-out of SMLP as a whole (discussed under Has the programme been implemented as intended?).

In relation to Columba 1400, it was commonly mentioned by school staff that the YPLA had helped to embed mentoring and leadership practices across their school. For example, one school’s participation in the Columba 1400 YPLA had helped embed more of a mentoring culture through changes to registration groups, encouraging discussions about values and shifting the approach to discipline. It was also noted, in the same school, that values-based approaches had been adopted among wider school staff (beyond those that took part in the YPLA). Columba 1400 staff also reported that they often receive feedback from teachers about the (sometimes unexpected) impact that the YPLA has had on them professionally in terms of how they view themselves as leaders.

In term of the Scottish Mentoring and Leadership Programme as a whole, given that the programme has not been rolled out at the scale and pace intended, and has reached considerably fewer schools and pupils than originally envisioned, we would conclude that mentoring and leadership have not yet been fully embedded across the Scottish school system as a whole.

Suggestions for improvement

The following are largely based on suggestions and comments made by participants interviewed for the evaluation.

At school/intervention delivery level:

• MCR should consider providing more information to parents/carers on the activities involved in the scheme and on the mentor that their child has been matched up with.
• Schools should aim to embed MCR Co-ordinators as much as possible (e.g. treating them as part of the pupil support team). (It may also be worth considering whether this would be beneficial for other specialist support).
• MCR should consider the following suggestions from mentors about ways in which their early stage experiences could be improved: an opportunity to have a one-to-one conversation with an experienced mentor; more use of breakout rooms in online training; training sessions in the evening so more people can attend; information about what school is like for young people nowadays; and raising awareness of peer resources available for mentors.

At the Scottish Government/strategic level:

• Given the very positive findings on the impact and Value for Money of MCR Pathways, review the funding model (including exploring alternative sources of funding) to enable MCR Pathways to be rolled out further and sustained, and encourage uptake in other Local Authorities/schools by publicising the evaluation findings.
• Conduct an evaluability assessment of the Columba 1400 YPLA to assess the feasibility of a quantitative impact and Value for Money assessment, which could then inform funding decisions.
• Scottish Government has been keen to understand ways in which any duplication of effort across skills/employment-related activities can be avoided. For example, some stakeholders compared MCR Pathways with other programmes and felt these interventions were available to similar cohorts of young people and had similar aims. We recommend Scottish Government undertake further review to ensure programmes are complimentary and continue to meet the needs of young people requiring specialist support.
• There are wider learnings about the importance of relational support and providing vulnerable young people with safe spaces (e.g. in small group settings with other young people or one-on-one meetings with trusted adults) to open up about their aspirations and the current concerns/difficulties which may be holding them back.
• The extent to which young people talked about their increased social confidence – as a result of both MCR Pathways and Columba 1400 YPLAs – was striking. They saw it as a hugely important, and sometimes transformative, capability in and of itself. But it was also the foundation for developing many other capabilities which they valued including better relationships with their peers and important adults (such as teachers) in their lives, and increased engagement in school and in their local community. It also helped improved their wellbeing (primarily through reducing anxiety). This suggests that effective interventions, focused on increasing social confidence and self-esteem, could have a hugely beneficial impact for many young people.

There are some further learning outcomes from the SMLP experience in relation to bringing different interventions/organisations together to deliver a joint programme:

• Consider first whether a joint programme is likely to add significant value, compared with the interventions being delivered separately, and how practical it is to bring them together.
• Allow enough time – several months at least – for relationships to be developed between delivery partners before the delivery stage.
• Actively facilitate the development of a shared vison and shared values for the joint programme.

Future monitoring

Building on the data monitoring that MCR Pathways and Columba 1400 already collects, the following are considerations for future data collection.

MCR Pathways

Theory of change alignment

MCR data collection tools currently capture information on a number of the ‘being’ and ‘doing’ capabilities in the SMLP theory of change (see Appendix 1). However, MCR could consider aligning data collection in each survey with more of the relevant capabilities. This would allow MCR to assess the impact of the intervention against the full range of intended outcomes from the programme. For example, for young people, the increased ‘social confidence’ ‘being’ capability could be measured by asking:

On a scale of 1 to 5 (where 1 is Strongly Disagree and 5 is Strongly Agree) to what extent do you agree or disagree with the following statement: “I feel confident in social situations”?

Baseline data

Gathering of baseline data could be improved by asking young people and mentors the same questions in the application form as in later surveys. This would help MCR to quantitatively measure the impact of the intervention. For example, the following question is included in the mentor end of year survey but could also be considered at the application stage in order to measure change in this over time:

Please rate your current confidence in the following areas (where 1 is Not at all Confident and 5 is Very Confident): Active listening; Empowering others; Helping someone set goals; Understanding the challenges young people face.

Other considerations

MCR could also consider adding questions to the mentor survey about support received from the Co-ordinator. For example:

How supported do you feel by your school's MCR Pathways Co-ordinator? - Very Supported / Supported / Neutral / Unsupported / Very Unsupported.

This could help identify whether Co-ordinators in different situations (e.g. in particular local authorities, those working across more than one school) are better placed to provide support.

Columba 1400

Data linkage

Recognising the challenges faced by Columba 1400 to date in linking data, introducing a unique pupil ID for all data collection points and procuring a data management system to facilitate data linkage would enable Columba 1400 to more easily measure the impact of the intervention over time.

Theory of change alignment

Columba 1400 data collection tools have the potential to capture qualitative information on a number of the capabilities in the SMLP theory of change (see Appendix 1). However Columba 1400 could consider further aligning quantitative data collection in the application form and evaluation form with the ‘being’ and ‘doing’ capabilities in the theory of change. For example, the increased ‘social confidence’ capability could be measured by asking:

On a scale of 1 to 5 (where 1 is Strongly Disagree and 5 is Strongly Agree) to what extent do you agree or disagree with the following statement: “I feel confident in social situations”?

Alignment with the ‘doing’ capabilities in both the applicaton and evaluation forms could also help Columba 1400 understand if the intervention is reaching young people who already display these cababilities and potentially enhancing them, or supporting young people to develop these capabilities for the first time. This could provide a stronger measure of the intervention’s impact. For example, the ‘stronger young person voice in schools’ capability could be measured by asking:

In the last school year, have you taken on any leadership roles? (This could be anything from being a team captain in PE, to helping a teacher, to organising something with friends) - Yes, often / Yes, a few times / Not really / No, not at all.