What and When to Evaluate: guide

What and When to Evaluate: A guide

Who is this guidance for and what does it cover?

Are you interested in finding out whether your policy intervention is working and achieving its expected outcomes? Or how to improve delivery on the ground? And whether you should be using evaluation to find out?

This guidance note was originally developed for Scottish Government policy and other professionals, but has been adapted to help other public bodies who wish to identify:

• what policies, programmes and projects (‘interventions’) should be formally evaluated;
• when to start planning, commissioning and undertaking an evaluation;
• how to determine the appropriate evaluation approach and budget.

This guidance covers:

• What evaluation is and why it is useful;
• The key questions evaluations can answer and the main types of evaluation;
• The types of interventions that are most appropriate to evaluate, with criteria to inform decision-making;
• Factors to consider when deciding on an evaluation approach and budget;
• When it is best to evaluate and plan for evaluation.

The guidance has been informed by other governments’ approaches to evaluation (see the references listed at the back of this document). AI (artificial intelligence) has been used minimally and was not a substantive part of this research process (i.e. only used for reference management).

What is evaluation and why evaluate?

The UK Government’s Magenta Book, the official guidance on evaluation, states that ‘Evaluation is the systematic assessment of the design, implementation and outcomes of an intervention.’ You may wish to read this guidance alongside, and in addition to, the Magenta Book.

Put more simply, evaluation is a set of tools or approaches to learn whether and how interventions are working and to demonstrate accountability in the use of public money.

Evaluation helps to ensure spending decisions are better evidenced through targeted use of both appraisal and evaluation.

Evaluation can also help to improve delivery on the ground, by finding out how interventions are working in practice, identifying problems and how to address them.

What types of questions can an evaluation answer?

There are three key types of questions that an evaluation can answer, and each can be addressed by one of three types of evaluation:

1. Does my intervention work or not, and is it achieving its intended outcomes and impact? – Impact evaluation

2. How is my intervention working on the ground, what problems are there with reach or delivery, and how can I improve delivery? – Process evaluation

3. Does my intervention offer value for money, and are the benefits it provides greater than the costs of delivery? – Economic or Value for Moneyevaluation (covering economic, financial, social, environmental and distributional costs and impacts)

Evaluation is only one part of the ROAMEF (rationale, objectives, appraisal, monitoring, evaluation and feedback) policy cycle that provides evidence to inform decision-making (see the Green Book). Whilst all of the stages of the policy cycle are connected, appraisal and monitoring are more closely interlinked with evaluation. For more information on their different roles, see the text box below and Annexes 1 and 2.

Appraisal, Monitoring and Evaluation

Appraisal is typically carried out before an intervention proceeds to assess the costs, benefits and risks of different options and to select the best one. An appraisal typically makes up one part of developing a proposal and is often part of a business case. It should also be used to inform any future evaluation.

Monitoring is the collection and review of data to see whether an intervention appears to be working. This can be used to track inputs, outputs and outcomes (see Annex 1). Whilst monitoring can tell us what is happening, it cannot tell us why it is happening, and importantly whether our intervention is causing the change observed in the data.

Evaluation is typically carried out after the intervention proceeds, to examine how an intervention has worked, ways to improve delivery, and if it succeeded in terms of achieving its intended outcomes. Evaluation can also be used as part of a piloting approach to see which option works best in practice before wider roll-out.

Inputs

• The resources needed to deliver the policy or programme

What we invest

Outputs

• The specific, direct things that the policy or programme delivers

What we get

Outcomes

• The results, effects or change produced by the policy or programme

What we achieve

Challenges: It should be noted that not all evaluations are straightforward, and evaluating some interventions may raise specific challenges and issues. For example, it is not always possible to fully assess Value for Money (VfM) or to attribute outcomes to specific interventions. Programmes may have both short and long-term outcomes which require different types of evidence, or data quality may be poor. It is helpful to explore these challenges at an early stage, as in some cases these challenges can be overcome, and to be clear about what types of questions the evaluation can answer.

Do I need to evaluate?

In an ideal world, all interventions would be evaluated to find out whether our interventions are working and/or how to improve delivery. However, in the real world of limited resources and time, it makes sense to focus efforts on which interventions are most appropriate to proceed to a full evaluation.

It is recommended that major investments, for example infrastructure like the reopening of the Borders Railway line (see text box below) or services like Fair Start Scotland (see page 14), that incur high costs be (appraised and) evaluated. Small-scale policy interventions with well-established evidence bases on effectiveness should not routinely be evaluated.

Case Study: Capital Investment – Reopening of the Borders Railway line (Criteria A, C and F)

The Scottish Government (SG) invested over £350 million to re-establish passenger railway services between Edinburgh through Midlothian to Tweedbank which opened in 2015. The 30 miles of new railway and seven new stations were expected to deliver major economic and social development opportunities. Asides from the economic appraisal/full business case that was undertaken prior to the decision to invest, a number of evaluations have been commissioned.

Process evaluation – to identify lessons learned from the delivery of the Borders Railway that could be applied in similar projects in the future.

Initial impact evaluations - year 1 and year 2 evaluations assessed the initial impact of the line (together with the important initial study to collect baseline data to enable a robust evaluation).

As the full impacts of such infrastructure investments only become fully apparent over time, discussions are ongoing about a longer-term evaluation to assess the extent of the realised economic and social benefits.

What criteria should I use to decide?

The following criteria are recommended to help decide whether an intervention should be evaluated. These criteria are in line with the principles for assessing whether to undertake an evaluation set out in the Scottish Government Evaluation Action Plan (2024-26) and other government guidance (see references). Answering ‘yes’ to any of these criteria increases the case for evaluation.

Evaluation criteria – Do I need to evaluate?

Is your intervention:

A. High financial cost (including one-off infrastructure investments);

B. A large-scale intervention with high population reach and/or impact;

C. Of strategic importance;

D. Of potential high and/or unknown impact on population groups with protected characteristics or other vulnerable groups;

E. High risk and/or high level of innovation and/or uncertainty;

F. High public profile /media /stakeholder interest or political sensitivity (evaluating would demonstrate accountability and/or support transparency and build trust);

G. Subject to an existing commitment to evaluate in either legislation or a published policy document;

H. Being introduced as part of a national crisis (for instance COVID-19) and would inform how to respond and/or be used to inform future crisis planning frameworks and review/Inquiry work.

Questions

Other questions to ask in which a ‘yes’ answer also increases the case for evaluation:

1. Is this an area with limited existing evidence and/or a knowledge gap?

2. Would the evaluation deliver evidence of sufficient quality?

3. Do you plan to use this evidence to inform your decision-making (for example, to improve your policy or as part of a Spending Review or budget process)?

4. Are there risks of not evaluating?

It should be noted that for criteria G, deciding not to evaluate when there is an existing legislative commitment to do so could leave your organisation open to legal challenge. The case studies in this guidance provide illustrative examples of evaluations and the criteria that applied, whilst the following sections provide more information on questions 1-4.

Existing evidence base (question 1)

To assess the extent of the existing evidence base, you can:

• Search the Evaluation Registry (a depository of all UK government and Scottish Government evaluations undertaken since April 2024)
• Search Scottish Government publications and/or your own organisation’s publications
• Consult with analysts/researchers within your organisation
• Consider the use of AI as part of a literature search and review (with a critical eye)

Finding out whether there are any existing systematic or literature reviews on the effectiveness of the intervention or similar interventions can also inform your decision on whether or not to commission an evaluation. Using the existing evidence base for your intervention in conjunction with a Theory of Change (see page 8)^[1] can be particularly useful to inform decision-making.

Even if this is an area with an established evidence base and previous evaluations, there may still be a case for a new evaluation if there has been:

• a significant change in the intervention or the context the intervention is operating in;
• a significant knowledge gap (e.g. in terms of impact on protected or vulnerable groups);
• significant risks and uncertainties relating to rollout of the intervention;
• issues with the quality and robustness of the existing evidence base;
• and/or a significant length of time has passed since the last evaluation.

This is particularly the case if any of the criteria from A-G above are met.

Quality of evidence from a new evaluation (question 2)

In relation to question 2 on whether an evaluation would deliver evidence of sufficient quality, analysts/researchers within your organisation should be able to advise you, using an evaluability assessment if required.

The UK Government’s Cabinet Office’s Evaluation Task Force has examples of projects that seemed more challenging to evaluate on first look, and the Scottish Government has published key principles and lessons for embedding evaluation more systematically into policymaking during times of rapid change or disruption.

Use of evidence (question 3)

Wherever possible, evidence from evaluations should be used to inform decision-making and not just undertaken to demonstrate accountability.

Evaluations can be used to:

• help decide whether to roll out a new intervention nationally;
• improve delivery of an existing policy;
• or to find out whether the intervention is achieving its outcomes, and to take difficult decisions to stop or significantly amend the intervention if it is not (see also the section on when to evaluate below).

Risks of not evaluating (question 4)

Evaluations can provide evidence to help manage and reduce the risk and uncertainty of developing and delivering government interventions. Choosing not to evaluate therefore also comes with risks, including:

• Failing to find out about unexpected adverse or costly impacts;
• Ineffective interventions might continue, wasting limited budgets; and,
• Missing opportunities to improve the design and delivery of the intervention and improving outcomes for the people of Scotland.

Theory of Change – how it helps appraisal, monitoring and evaluation

A Theory of Change (TOC) sets out how an intervention, or set of interventions, is expected to work (in theory) to achieve intended outcomes. Starting at an intervention’s inputs and activities, a TOC sets out the steps and pathway (or logic) through which an intervention is expected to deliver its outputs and outcomes (short, medium and long-term outcomes).

A Theory of Change is typically produced through workshops with stakeholders and/or by bringing together the existing evidence base. A TOC should also set out the assumptions made, the wider contextual factors which may impact delivery or outcomes, and ideally the quality and strength of the evidence supporting the pathways. A TOC can therefore highlight dependencies, why the intervention might not work and where the risks and uncertainties lie.

A TOC helps an appraisal set out the rationale for government intervention, and helps state the assumptions about a proposed intervention and to test its logic. A TOC can help foster a shared understanding of how change will happen.

A TOC also helps identify what inputs, outputs and outcomes (see also Annex 1) need to be measured for the intervention, i.e. to decide on what data needs to be collected and/or brought together for monitoring. By identifying evidence gaps, a TOC can also inform and help to structure the evaluation approach.

As Figure 1 on page 9 shows, a TOC should visually demonstrate the route from Inputs (resources committed and activities undertaken) to Outputs (what is delivered or produced – ‘mechanisms for change’), Outcomes (early or medium term results) and Impact (long-term results). It should also capture Assumptions (e.g. around how the programme will work), Moderating Factors (which may affect the strength and direction of the process) and any supporting activities.

The Scottish Government’s Distress Brief Intervention (DBI) programme evaluations (see case study on page 10) were both underpinned by theories of change (see DBI Appendix A, under 18s supporting document).

Ultimately, depending on the scale, impact and profile of the intervention, this could result in reputational risk and a loss of stakeholder and public trust.

If you have worked through the criteria and supplementary questions above and have concluded that there is not a sufficient case to proceed to a formal evaluation, you may still want to consider other ways of reviewing and improving your intervention. See the text box on page 11 for more information.

Case Study: Distress Brief Intervention (Small spend innovative intervention with vulnerable groups – Criteria D and E)

Distress Brief Intervention (DBI) is an innovative way of supporting people who are in distress. The Scottish Government decided to run a pilot of DBI in Aberdeen, Inverness, Lanarkshire and Scottish Borders between 2016-2021, and also conducted a process and impact evaluation at the same time. This was to see if the intervention worked as intended and to learn lessons to improve delivery of DBI prior to deciding on any national rollout. The pilot evaluation found that DBI was successful in offering support to those in distress.

In 2020, the SG extended DBI across the whole of Scotland during the COVID-19 pandemic. A separate evaluation (criteria H – national crisis) concluded that despite COVID-19 and associated restrictions, the extended service adapted successfully and provided telehealth support to individuals presenting with mild-moderate distress.

Given the decision to extend DBI to those aged 16-18, a further evaluation of the under 18s element was undertaken too. It concluded that DBI could be an effective intervention for use with young people though was not at a stage for national rollout.

The Scottish Government spent £193k on the pilot evaluation, 5.7 per cent of the original DBI pilot cost (£3.4 million, 2016-21). Subsequent evaluations cost £75k (2022) and approximately £150k (2025) respectively, a small percentage of the annual funding for DBI delivery (£2.9 million in 2025/26).

What scale and type of evaluation(s) is best and what budget is appropriate?

There are some key factors to consider when deciding on a proportionate evaluation approach and scale, and the budget for this activity.

Deciding on an evaluation approach will depend primarily on the key questions of interest, which in turn will relate to an understanding of the existing evidence base (including monitoring data), and the feasibility of an evaluation.

The appropriate and proportionate scale of an evaluation also relates to some of the key criteria above. Interventions with any of the following factors will justify a higher budget than interventions that do not meet any of these criteria:

A. A high financial cost (including one-off infrastructure investments);

B. A high reach and/or impact (scale of the intervention);

C. Strategic importance;

D. A high level of risk and/or innovation;

E. Potential high and/or unknown impact on groups with protected characteristics.

Alternatives to formal evaluation

There are a number of useful alternatives to formal evaluation. For example, you could review and improve the monitoring data you collect to better understand delivery and impact, and/ if you do not have a Theory of Change for your policy, you may want to hold workshops with stakeholders to produce one in order to better understand your intervention(s). This would also allow you to check the logic of your assumptions and the realism of the pathway through which outcomes are expected to be achieved and monitored.

Workshops with stakeholders can also be used to hear from those closer to delivery ‘on the ground’ about what is and isn’t working, or could be improved. Quality improvement techniques can also be used to test and improve the delivery and impact of existing interventions.

If you are at the early stages of the policy development process, you may wish to adopt a Test and Learn approach. This is an iterative approach to the development of a new intervention. It involves several rounds of ‘testing and learning’ on a small scale that maximises the chance of achieving the intended outcome/s on wider rollout. This approach will help to build the evidence base and confidence in moving towards a proportionate evaluation once rolled out (see Test and Learn Annex in Magenta Book).

The scale of evaluation required should also take account of available resources and organisational priorities, as well as ensuring that the allocated budget can deliver an evaluation of high quality. Consideration should also be given to the contribution to the wider evidence base and the potential to enable analysis or inform decisions on other programmes or initiatives, both now and in the future.

You should also consider what is feasible and proportionate to evaluate within the available time and resource. If this is a complex question, an evaluability assessment^[2] may be required to determine the parameters of the evaluation.

The Cabinet Office’s Evaluation Task Force recommends that 2-10% of the intervention’s total budget be set aside for evaluation, with higher budget projects being at the lower end of the range (as a lower % of a higher budget will give sufficient funding for a high quality evaluation). The budget for evaluation should also take account of the level of the intervention’s scale, risk and uncertainty.

Type of evaluation

In deciding on the evaluation approach and scale, it is helpful to return to the three key evaluation types and questions they can answer (see page 4).

Evaluation, as well as the development of the intervention, its appraisal and monitoring, should also ideally be underpinned by a Theory of Change (pages 8 and 9).

In many instances, a combination of evaluation types are undertaken, for example a process and impact evaluation undertaken together, or in a sequence depending on when the information is needed for decision-making. See the Fair Start Scotland case study on page 14 for an example.

Your analytical colleagues within your organisation can help you think through what are the most important evaluation questions to address for your intervention, and the timing of them, taking account of any budget and/or other constraints. They will also guide you on evaluation methods.

When should I evaluate?

Regardless of when you think you might evaluate, evaluation should be planned early, i.e. built into the appraisal considerations, and intervention development and design from the outset. This is because early policy decisions can significantly limit the type, robustness and usefulness of the evaluation later on. For example, baselines and baseline data, as well as any comparisons or control groups, require to be set up from the outset of policy development.

Early planning avoids problems later on and can hugely improve the quality of your evaluation and therefore your decision-making. The Green Book states that if there is an appraisal of the intervention, evaluation planning should start with analysts at this point. In any event, it is a good idea to involve your analysts/researchers as soon as possible during the development of your intervention and before key decisions that impact on the evaluation are made.

Evaluation is potentially relevant at the design, implementation, delivery, and review stages of an intervention’s life cycle. However, deciding when is best to plan, commission and undertake an evaluation for your intervention depends on its own particular context, aims and levels of spend.

For example, the Scottish Government’s policy team for Distressed Brief Intervention (DBI) did a small scale pilot in four areas and commissioned an impact evaluation to see if and how the intervention worked before deciding whether to roll it out across all of Scotland. Alternatively, the Early Learning and Childcare Team decided to conduct a pilot to evaluate the impact of implementing a policy to ensure all children whose parents/carers opted to defer their entry to Primary School remained eligible for Funded Early Learning in their deferral year. This allowed the team to test the impacts on capacity within nurseries and understand the effectiveness of communication with parents and carers.^[3]

Looking at the evaluation criteria (page 6) can also be helpful here. For example, an intervention with high spend and strong public interest, and a smaller existing evidence base, would have a strong case for an impact evaluation to be carried out sooner rather than later. This would avoid continuing to spend limited resources on an intervention that may or may not work.

In order to maximise the usefulness and impact of evaluation evidence, the timing of evaluations should also be linked to decision-making. This may involve some flexibility in terms of the timing of both decision-making and the evaluation itself. Interim reports on emerging evaluation findings can sometimes help to bridge any gaps. However, if the timing of any decisions does not align with the delivery of the evaluation findings, and they cannot be made to fit, you should consider whether to evaluate the intervention at this point or to delay if required.

Finally, all evaluations, whether carried out externally by an independent contractor (the majority of Scottish Government evaluations), or carried out by in-house analysts directly, usually have a minimum lead-in period for planning of around 3 months, plus time to commission if opting for an external evaluation. This is because, amongst other things, evaluations require a clear research specification for the work. Again, your organisation’s analysts/researchers can advise you on their timescales (also resource dependent) and the best option for commissioning an evaluation.

When to evaluate - key points

• Early engagement and planning for evaluation with your organisation’s analysts / researchers at appraisal and/or intervention design stage (baseline data, comparator and control groups)
• Review criteria for evaluation and timing of key decisions to inform your decision on when to evaluate
• Note the minimum lead in time for planning an evaluation of around 3 months plus time to commission if opting for external evaluation.

Case study: Fair Start Scotland (use of process, impact and value for money evaluations and monitoring data – Criteria A, B, C, D and H)

Fair Start Scotland (FSS) was a voluntary employability support service set up to provide personalised support to eligible people in Scotland to help them find and stay in work. With an annual running cost of between £14 and £28 million per annum, FSS was evaluated through a series of evaluations over its 6 year operating period from April 2018.

Key evaluation outputs included:

2019 implementation review of the first 6 months of the programme;

Process and impact evaluation reports for Years 1-6;

An economic/value for money evaluation of Years 1-3.

The Scottish Government also collected and published monitoring statistics for Fair Start Scotland (FSS).

Evaluations of Fair Start Scotland (FSS) showed that it improved labour market outcomes and had a positive impact on the wellbeing of participants. Overall satisfaction with the service was high and the vast majority of participants felt they were treated with dignity and respect. Economic analysis found that FSS offered good value for money.

The evaluation evidence was used for decision-making, including changing aspects of service delivery such as the inclusion criteria for joining the service.

Evaluations also found that women and individuals from ethnic minority groups were less likely to rate the FSS as highly as males and white individuals. Therefore, in collaboration with Fair Work colleagues, the Scottish Government funded a post with CEMVO to share their expertise with FSS and ensure that ethnic minority participants receive the required level of support to help them progress into and sustain employment. Wider learning from Fair Start Scotland (FSS) has also informed the next stage of delivery of Scotland’s funded employability provision through the No One Left Behind approach.

The total cost of Fair Start Scotland (FSS) evaluations was £435k. Whilst this may sound high, it is a small percentage of the annual running costs (less than 1 per cent each year) and an even lower percentage of the total funding for FSS over its six year lifespan.

Challenges in evaluating Fair Start Scotland included the lack of a control /comparator group (of people who did not receive help from FSS) and linked earnings data to quantify the programme’s impact more robustly.