What and When to Evaluate: guide

Annex 1. Monitoring vs evaluation and inputs, outputs and outcomes

Monitoring and evaluation data provide evidence to inform decision-making as part of the policy cycle, and good quality monitoring data can be used as part of an evaluation. However, there are some key differences between monitoring and evaluation and the robustness of the evidence they provide for decision-making.

To explain this, it is useful to understand the difference between an intervention’s inputs, outputs and outcomes, and the relationship between them (see Figure A1.1),

Inputs

• The resources needed to deliver the policy or programme

What we invest

Outputs

• The specific, direct things that the policy or programme delivers

What we get

Outcomes

• The results,effects or change produced by the policy or programme

What we achieve

Inputs, outputs and outcomes are also the basic building blocks of theories of change (or ‘logic models’) which set out how the intervention is expected to work and the pathway (or logic) from inputs and activities to outcomes (see the text box on page 8 for how theories of change can inform appraisals, monitoring frameworks and evaluations). In summary:

• Inputs are the resources needed to deliver the intervention, in other words what we invest, e.g. money for staff;
• Outputs are the specific, direct things that the intervention delivers, or what we get, e.g. more and earlier outpatient appointments;
• Outcomes are the results, effects or change produced by our intervention, in other words what we achieve, e.g. improved cancer survival rates.

Monitoring data can be used to track or monitor how an intervention is expected to create change by tracking inputs, outputs and outcomes.

Monitoring is useful as it allows us to track whether we appear to be heading in the right direction without commissioning a full evaluation which can take time and is usually more expensive.

However, whilst monitoring can tell us what is happening with our inputs, outputs and outcomes, it cannot tell us why it is happening, and most importantly whether it is our intervention (or set of interventions) which is causing the change observed in the outcome data.

Monitoring does not allow us to understand the relationship between inputs, outputs and outcomes and it does not allow us to attribute any observed changes in outcomes to our interventions. In other words, using monitoring data tells us nothing about causality so we do not know whether or to what extent our policy is working or not, and how or why.

For example, an outcome measure could be heading in the wrong direction, such as child poverty rates increasing. However, an increase in child poverty rates could be because of an increase in energy and food prices whilst wages stay at the same level. Evaluation, combined with high quality monitoring data, can help us understand whether and how much our policies to tackle child poverty are working to help keep any increase in child poverty to a minimum during a cost-of-living crisis.^[4]

A further example of how monitoring data alone can be insufficient – and why evaluating Fair Start Scotland was a good decision (see case study above) – can be seen in how employment and unemployment rates generally shift in line with how the overall economy is fairing. Say the Scottish Government set up an employment support service for people who are disabled and people at risk of long-term unemployment. In an expanding economy, outcome monitoring data could show unemployment rates declining amongst both groups leading us to think that our employment support service is working well.

However, this could be down to the economy improving and more jobs being available rather than our assistance and support package for the unemployed being effective and helping those that it should. Only a well-designed impact and process evaluation would give us evidence on the effectiveness of the service and how it could be improved. The case study (page 14) on Fair Start Scotland (FSS) - Scotland’s employment support service until 2024 - shows how the Scottish Government used different types of evaluation, and monitoring data, to improve the service FSS offered and maximise its outcomes and impact.

Further reading

Scottish Government analysts and policy colleagues developed a monitoring and evaluation framework for the Cancer Strategy for Scotland 2023-33 and the Cancer Action Plan 2023-26.