Public service reform strategy - theory of change, and monitoring, evaluation and learning framework: update

2. Monitoring, Evaluation and Learning Framework

Introduction

This section describes the development of a Monitoring, Evaluation and Learning (MEL) Framework to measure the progress of the Scottish Government’s Public Service Reform (PSR) Strategy over a 5-year timeframe from 2025-2030.

The process of developing a Theory of Change for the PSR Strategy – as outlined in the previous section – has been undertaken to inform and structure our approach to monitoring and evaluating the Strategy, including identifying suitable indicators to measure progress.

The PSR Strategy was intentionally wide in scope to tackle the challenges it intends to address. It also emphasised the importance of monitoring and evaluating impact at the system-wide level. We are developing this Monitoring, Evaluation and Learning Framework to be also wide in scope, and to allow us to effectively measure and understand the impact of actions within each of the Foundations and Pillars of the Strategy.

Together with the Theory of Change, this section provides an overview of the approach to be taken in monitoring and evaluating the Strategy, and also for learning. This includes:

• Identifying the long-term outcomes the PSR Strategy intends to achieve, and the mechanisms of change that will enable this;
• Providing an example set of indicators we are considering for the long-term outcomes and for one of the four mechanisms of change.

Further work will take place in the coming months, to complete the process of indicator identification and assessment. An update to this Monitoring, Evaluation and Learning Framework and baseline assessment will be published by autumn 2026.

1. Alignment with other measurement frameworks

To reduce duplication and improve alignment of measurement activity, it is important our approach to MEL reflects and recognises other SG measurement frameworks that are currently in development or are being reformed. This includes the Verity House Agreement (VHA) Monitoring Accountability and Assurance arrangements which has overlaps for what we propose for monitoring and evaluating the PSR Strategy.

In addition the Scottish Government is currently reviewing Scotland’s National Performance Framework (NPF) and has recently shared an update on progress to the Scottish Parliament Finance and Public Administration Committee (and all other Committees). This update outlines a new draft model for the NPF based on international good practice, adapted to fit the Scottish context. We propose drawing upon indicators from the refreshed NPF to inform the PSR Strategy MEL Framework.

As this work is ongoing and in the case of the NPF, is the subject of currently ongoing external engagement, we feel that it would not be appropriate to finalise the PSR Strategy’s Monitoring, Evaluation and Learning Framework until these other exercises are completed. We also intend to align this Framework with the development of the measurement approach for the Scottish Government and COSLA's Population Health Framework.

2. Measuring the Public Service Reform Strategy

Referring closely to the PSR Strategy ToC we propose that the Framework will use a range of indicators to measure progress across the Public Service Reform (PSR) Strategy. These indicators will provide specific and measurable evidence of change across the Foundations and Pillars of the Strategy and will enable us to measure progress against its outcomes, as set out in the Theory of Change. The Theory of Change is the basis for how we will structure this framework, in order to provide as complete a picture of progress as possible, given the strategy’s wide scope.

The indicators to be used by this Framework will prioritise the use of existing data sources, or data sources that are already in development. This approach is intended to minimise duplication, reduce administrative burden, and ensure alignment with other measurement frameworks described above.

We acknowledge that no measurement framework will provide perfect measurement of all variables that are potentially of interest, particularly where the scope of the actions and activities are broad, as they are here. To address this we will seek to supplement quantitative indicators with qualitative evidence where possible, to provide a more complete picture of progress. Where the data sources allow, we will look at progress against intended outcomes for different demographic and geographic groups, including protected equality characteristics.

The PSR Strategy makes specific reference to some ways in which the Scottish Government will measure success under each of the three Pillars:

• Pillar 1: Prevention We will measure preventative spend by Government and track that the proportion of spend on prevention increases and the resultant spend on acute/crisis decreases.
• Pillar 2: Joined Up Services We will:
  • Measure the impact of services against required outcomes.
  • Measure cost reductions and cost avoidance generated by service change, and the proportion of those savings invested in further change.
• Pillar 3: Efficient Services Over the next 5 years we will reduce annualised Scottish Government and public body corporate costs by £1 billion.

The Theory of Change sets out four Long-term outcomes, that reflect the key aims of Public Service Reform, and describe the change that we want to see in 5 or more years’ time:

• Long-term outcome 1: Improving people's outcomes and experience of services
• Long-term outcome 2: Improving community services
• Long-term outcome 3: Improving service delivery and financial sustainability
• Long-term outcome 4: Developing the public sector workforce

The Public Service Reform Strategy has 4 Foundations and 3 Pillars, with the foundational workstreams shaping work across the other 14 workstreams. The mechanisms of change and long-term outcomes set out in the Theory of Change reflect the Foundations and Pillars respectively, and by focusing on these two areas we will monitor and evaluate the two main components of the Strategy.

The relationship between the different components of the PSR Strategy and the Theory of Change and Monitoring, Evaluation, and Learning Framework is set out visually in Figure 3.

The long-term outcomes reflect what the Pillars of the Strategy intend to achieve. However, workstreams contained within each of the Pillars can contribute to more than one outcome, and as a result the Pillars do not have a direct one-to-one relationship with the outcomes.

A diagram showing the relationship between the Foundations and Pillars of the Public Service Reform Strategy, and the Monitoring, Evaluation and Learning Framework. The Foundations will be monitored and evaluated through a focus on the Mechanisms of Change, and the Pillars will be monitored and evaluated through a focus on the Long-term outcomes, as set out in the Theory of Change.

The measures of success described in the Strategy under each of the Pillars will be reflected as indicators for the long-term outcomes, supplemented by other measures.

To track progress across the four long-term outcomes, we will identify and use a set of indicators for each of the outcomes, including those that relate to the Pillar measures of success given in the Strategy.

Table 1 provides examples of the indicators which will be used for each. Over coming months we will continue to assess and add to these indicators with further consideration of indicator properties (for example, closeness to outcome concept being measured, population coverage, frequency of publication, sensitivity to change, availability of disaggregated data for demographic groups, and so on).

Table 1. Potential indicators for Public Service Reform Strategy Long-term outcomes

Long-term outcomes (from Theory of Change)

1. Improving people's outcomes and experience of services

Description

People in Scotland will be able to access efficient, good quality and person-centred public services. This will improve people’s lives, improve outcomes and contribute to reducing inequality in Scotland.

Potential indicator/indicator sets

Long-term indicator 1A: Satisfaction with local health, schools, and other services.

Long-term indicator 1B: Public service outcomes.

Long-term outcomes (from Theory of Change)

2. Improving community services

Description

Scotland’s communities will have more power over how services are designed and delivered in their local area, with an increase in local decision-making.

Potential indicator/indicator sets

Long-term indicator 2A:

Level of community influence and participation in public services.

Long-term outcomes (from Theory of Change)

3. Improving service delivery and financial sustainability

Description

Service providers across Scotland will work together in a cost-effective, efficient and preventative way - with a focus on learning and improvement.

We will reduce Scottish Government and public body corporate operating costs by £1 billion by 2030

Potential indicator/indicator sets

Long-term indicator 3A: Proportion of spend on prevention increases and the spend on acute/crisis decreases.

Long-term indicator 3B: Measures of cost reduction and cost avoidance generated by service change, and the proportion of those savings invested in further change.

Long-term indicator 3C: SG and public body corporate cost reduction.

Long-term outcomes (from Theory of Change)

4. Developing the public sector workforce

Description

This workforce will be smaller and more sustainable, with staff who have the right skills for the future.

Potential indicator/indicator sets

Long-term indicator 4A: Size of public sector workforce.

Long-term indicator 4B: Evidence on public sector workforce skills.

The Theory of Change also provides an explanation of how the activities in the Strategy are intended to create change and therefore contribute to the long-term outcomes set out above.

These ‘mechanisms of change’, as set out in the Theory of Change and set out in the accompanying ToC narrative in the first section of this document, are about the processes, or underlying drivers, that explain how and why our activities and programmes may enable change (not the change itself). Monitoring and evaluating these aspects will enable us to understand whether the mechanism is operating and if the theory or ‘logic’ of the Strategy is working as intended, with the right people engaged, and the appropriate activities occurring.

The Theory of Change contains four main mechanisms of change, with each containing 2-3 sub-themes.

• Mechanism of change 1: Culture and workforce
  • Leadership
  • Workforce capability
• Mechanism of change 2: Enabling Infrastructure, Digital and Data
  • Budgeting
  • Data availability
  • Technology for service redesign
• Mechanism of change 3: Ways of working
  • Partnership
  • Simplification
  • Integrated working
• Mechanism of change 4: People’s experiences, trust and fairness
  • Reducing inequality
  • Community involvement
  • Trust

These change themes describe the human and organisational dynamics which will collectively contribute to achieving the long-term outcomes of the Public Service Reform Strategy. We will also measure these ‘mechanisms of change’ through a set of indicators as a means to assess the extent to which these changes are happening as expected.

In summary, the Monitoring, Evaluation and Learning Framework will measure progress against two aspects of the programme through a series of indicators (see Table 2 below):

Table 2. Summary of how the Monitoring, Evaluation and Learning Framework is intended to measure progress

Mechanisms of change >

Indicators relating to each mechanism of change:

Long-term outcomes

Indicators relating to each long-term outcome:

Mechanisms of change >

1. Culture and workforce

1.1. Leadership

1.2 Workforce capability

2. Enabling Infrastructure, Digital and Data

2.1 Budgeting

2.2 Data availability

2.3 Technology for service redesign

3. Ways of working

3.1 Partnership

3.2 Simplification

3.3 Integrated working

4. People’s experiences, trust and fairness

4.1 Reducing inequality

4.2 Community involvement

4.3 Trust

Long-term outcomes

1. Improving people's outcomes and experiences of services

2. Improving community services

3. Improving service delivery and financial sustainability

4. Developing the public sector workforce

Mechanisms of change >

30 to 40 indicators in total

Long-term outcomes

6 to 10 indicators in total

Table 3, below, sets out an example for Mechanism of change 4: People’s experiences, trust and fairness, including potential indicators and indicative data sources. At present, there are gaps in these indicators. For example, on our first assessment, there is more readily available, regularly reported data on healthcare than some other kinds of public services. Further, concepts such as trust have multiple dimensions. Measuring only trust in public services will miss how reform is experienced across the system.

Table 3. Mechanism of Change 4. People’s Experience, Trust and Fairness – Potential indicator examples

Sub-theme

Community Involvement

Indicator

Percentage of respondents who agree “I can influence decisions affecting my local area”

Data source

Scottish Household Survey

Frequency of reporting

Annual

Sub-theme

Community Involvement

Indicator

Operation of legislation/other mechanisms to support community participation (e.g. Community Empowerment Act (2015)

Data source

Community Empowerment Act (2015) Administrative data (potential)

Frequency of reporting

TBC

(Potential indicator)

Sub-theme

Community Involvement

Indicator

Responses to Scottish Government Consultations on public services

Data source

Citizen Space

Frequency of reporting

TBC

(Potential indicator)

Sub-theme

Accessing Services

Indicator

Percentage positively rating arrangements for speaking to a doctor at

GP

Data source

Healthcare Experience Survey

Frequency of reporting

Biannual

Sub-theme

Accessing Services

Indicator

Percentage positively rating arrangements for speaking to a mental health professional at

GP

Data source

Healthcare Experience Survey

Frequency of reporting

Biannual

Sub-theme

Accessing Services

Indicator

Percentage that believes anyone can access the justice system if needed

Data source

Scottish Crime and Justice Survey

Frequency of reporting

Biannual

Sub-theme

Trust

Indicator

Percentage of people who trust a great deal, or tend to trust Local Government

Data source

Scottish Household Survey

Frequency of reporting

Biannual

Sub-theme

Trust

Indicator

Percentage of people who trust a great deal, or tend to trust the education system

Data source

Scottish Household Survey

Frequency of reporting

Biannual

Sub-theme

Trust

Indicator

Percentage of people who trust a great deal, or tend to trust the health system

Data source

Scottish Household Survey

Frequency of reporting

Biannual

Sub-theme

Trust

Indicator

Percentage of people who trust a great deal, or tend to trust the justice system

Data source

Scottish Household Survey

Frequency of reporting

Biannual

Sub-theme

Trust

Indicator

Percentage of people who trust a great deal, or tend to trust the police system

Data source

Scottish Household Survey

Frequency of reporting

Biannual

Sub-theme

Trust

Indicator

Percentage of people who say that a problem with funded childcare is the lack of providers they trust to provide a quality service

Data source

Scottish Household Survey

Frequency of reporting

Biannual

Engagement with Scottish Government workstream leads and external partners will look to identify additional data sources to fill these gaps and develop a full list of indicators, along with appraising their suitability for this purpose.

Next steps

We are in the process of engaging with Scottish Government workstream leads to agree appropriate indicators to measure progress, and to identify data sources. As described previously, we expect this will involve a combination of quantitative data drawn from existing sources, alignment with other measurement frameworks, and qualitative data (such as interviews or case studies) to understand experiences and processes.

This timescale will allow for further stakeholder engagement, and alignment with monitoring and evaluation approaches being taken across the Scottish Government for related work including the refreshed National Performance Framework, Verity House Agreement Accountability and Assurance arrangements, and the development of the measurement approach for the Population Health Framework.

We intend to carry out targeted engagement with partners and stakeholders to the Public Service Reform (PSR) Strategy on the choice of indicators used in the final indicator set, using the Theory of Change and other material in this technical update paper as a basis.

Following this work, an updated version of the PSR Monitoring, Evaluation and Learning Framework and indicator set will be published by autumn 2026. This framework will form part of a wider range of data and evidence used by the Scottish Government to monitor progress on public service reform, including governance and evaluation processes being established within individual workstreams.

The framework and indicator set will be reported on an annual basis, with the first report published by April 2027. This will provide a narrative of progress across the strategy built on the indicator set. It will identify good practice, delivery challenges and risks, and actions to address these. It will be used to support learning, improvement and prioritisation.

On the two-year and five-year anniversary of the PSR Strategy being published (June 2027 and 2030) we plan to conduct a more in-depth evaluation of the delivery and the impact of the Strategy. This evaluation activity will draw upon the data collected as part of the PSR Monitoring, Evaluation and Learning Framework annual reporting, and be supplemented with additional data such as interviews with those involved in the delivery of the Strategy.