Leadership Board: First meeting - PHS - Monitoring and Evaluation Paper

Monitoring and Evaluation for Scotland’s Mental Health & Wellbeing Strategy

A summary of advisory work to aid decision- making and prioritisation

The Scottish Mental Health and Wellbeing Strategy (the Strategy) was published in June 2023. Evaluation of the Strategy – or components of it – will be key to understanding whether it has contributed to its vision of ‘a Scotland, free from stigma and inequality, where everyone fulfils their right to achieve the best mental health and wellbeing possible’.

Public Health Scotland (PHS) was asked to assess the feasibility of evaluating the Strategy. This short briefing summarises the purpose, findings, and recommendations from that process. It is intended to guide decision making about monitoring and evaluation (M&E) of the Strategy.

Purpose of the exercise

The purpose of the assessment was:

• To support shared understandings of how the Strategy may achieve its intended outcomes, and how these may be assessed over time.
• To provide a preliminary understanding about approaches to M&E which are feasible and proportionate to the aims of the Strategy.
• To identify potential options for M&E that support learning and inform plans for the longer-term.

The approach taken

To complete this initial assessment, PHS:

• Conducted a series of engagement sessions with key stakeholders (SG, COSLA, and PHS staff) working in areas relevant to the Strategy.
• Completed a rapid review of the evidence in two key areas of the Strategy.
• Scoped available data and summarised key gaps.
• Explored potential approaches to M&E of the Strategy.

Following this process PHS suggested a series of next steps.

Framing an approach to monitoring and evaluation

Taking this systematic approach at an early stage has helped to identify key points.

Due to its scale and complexity, it will not be possible to carry out a comprehensive evaluation of the entire Strategy; this would not be feasible or proportionate and would not be meaningful. However, elements of the Strategy can be evaluated, and learning brought together within a broad framework. To do this further work can help to identify and prioritise key areas of the Strategy for evaluation activity.

Evaluation of the Strategy presents two key challenges: 1) measuring expected outcomes and 2) attributing these outcomes to the actions within the Strategy. Given the complex, dynamic policy context in which the Strategy is being implemented, directly attributing outcomes to the Strategy is not feasible or realistic. Taking a theory-based approach can account for this complexity (Appendix 1). It will enable us to:

• Draw conclusions about how different elements of the Strategy contribute to any changes in outcomes, how, and for which people.
• Use a diverse range of approaches and methods to M&E for these different elements of the Strategy (e.g. process, impact, economic evaluations using appropriate methods).
• Provide an overarching M&E framework that includes learning about delivering short term outputs or products, implementation and delivery, as well as keeping a focus on the longer-term outcomes for which preventative action is required.
• Supporting a long-term view to identifying and measuring the outcomes achieved for people’s mental health and wellbeing.
• Support a collective understanding of the progress and impact of the Strategy within the wider system, and within the context of many other factors that influence mental health and wellbeing outcomes. This is a useful approach to evaluating policy in complex, real-world scenarios and would help to build a narrative about the progress and impact of the Strategy over time.

To support this first stage of work:

• A draft high-level ‘theory of change’ was created to visually show how the strategy is expected to contribute to long term outcomes (Appendix 2).
• Sequenced illustrative evaluation questions were drafted, based on the outcomes in the Strategy, to illustrate the types of evaluation questions that may be required at different stages of implementation through to understanding potential impact.
• A scoping of available data included mental health outcomes, some wider determinants of mental health outcomes, and some services and supports. Critical gaps exist, particularly for protected characteristics and other demographics, non-NHS service use, people’s experiences and understanding, and decision-making for policy.
• Stakeholders were interested in minimising additional data collection and reporting asks and were keen to move towards an impact-focussed evaluation that addresses the differences made to people’s lives. This was considered to be aligned to the Verity House Agreement principles.

Recommended next steps

The Strategy Leadership Board is asked to either agree or provide comment on the recommended next steps.

To facilitate the next stage of M&E work, the Leadership Board is asked to provide agreement that:

1. The proposed approach to developing a framework for M&E work should continue, to enable the contribution of the Strategy to be understood over time (see Appendices 1 & 2).

2. A focus on the medium and long-term impacts of current and future actions is crucial to understand the contribution of the Strategy over time (i.e. the differences made to people and communities), in addition to short-term M&E.

3. A Monitoring and Evaluation Sub-group should be established as part of the governance structures for the Strategy. This is needed to broaden specialist and technical expertise to develop M&E plans and report to the Leadership Board as required. It would also work closely with a broad range of relevant stakeholders to develop plans.

4. The Monitoring and Evaluation Sub-group should have a lead role in advising on key areas of the Strategy for M&E. This should be based on the Strategy priorities and outcomes, rather than solely on existing availability of data, indicators, and evidence. A prioritisation process would be developed to guide this process, ensure transparency in decision making and reduce the potential for duplication of efforts. It will also be crucial to ensure that limited resources for M&E are well utilised and that M&E is proportionate to the resources available, and the knowledge generated.

5. To achieve point 4 above, work with stakeholders is required to understand which Strategy outcomes are of most interest, which questions are the most important for M&E of the Strategy, what data exists that may be suitable to address these questions over time, where gaps exist, and how resources might be directed to fill those gaps (e.g. with new evaluation).

6. For the key areas identified for M&E:

a. Further evidence review work will connect evidence with theory and be used to identify areas where there are gaps in knowledge that might require to be evaluated.

b. Data mapping will be completed in preparation for the development of detailed M&E plans, to identify potential gaps and ways to address these.

c. The use of internal SG, Local Authority (and other partners) routine data will be explored for potential use in any early process evaluation to understand whether and how progress is being made in the Strategy’s implementation.

For further information on this paper and the technical report please contact phs.evaluationteam@phs.scot