Strategic commercial interventions: assurance playbook

Evaluation Guidance

What is it?

Evaluation is a systematic assessment of the design, implementation and outcomes of an intervention. It involves understanding how an intervention is being, or has been, implemented and what effects it has, for whom and why. It identifies what can be improved and estimates its overall impacts and cost-effectiveness.

There are two main types of evaluation: outcome evaluation and process evaluation. Outcome evaluation provides an assessment of whether the intervention worked as expected – to what extent the expected outcomes were realised and at what cost. In general terms an evaluation should:

• Assess whether the intervention met its objectives.
• Compare actual outcomes observed from the intervention with the outcomes expected for the counterfactual (which is what is likely to have happened in the absence of the intervention) to determine the extent to which the changes observed can be attributed to the intervention, as opposed to other external factors.
• Provide an assessment of whether the intervention has provided value for money to date.
• Process evaluation provides an assessment of how well the intervention has been implemented and provides learning points to both improve the current intervention and to inform the implementation of future strategic commercial interventions. It may also be known as ‘lessons learned’. Specifically, it considers whether the intervention followed correct processes and procedures (audit-type check); and draws on feedback from key internal and external stakeholders involved in the intervention to assess how well the intervention was implemented.

If an objective of the intervention was to return the business to commercial viability so that it could operate without government support in future, then an assessment of the business’ financial health and commercial viability post-intervention compared to pre-intervention should be assessed as part of the evaluation. This guidance sets out some specific considerations to assess business health.

This guidance, which is aligned with the Business Investment Framework within the Scottish Public Finance Manual (SPFM) sets out the purpose of evaluation and its role in the policy cycle (figure one). It also covers the differences between a ‘stocktake’ and process and outcome evaluations and highlights the importance of considering proportionality when undertaking them.

What is its purpose?

It is essential that public funds are spent on activities that provide the greatest benefits to society and that they are spent in the most efficient way. To help assess this the Scottish Government (SG) requires a clear process to evaluate its strategic commercial investments in a robust, evidence-based and objective-led way, that can be consistently applied across interventions.

The purpose of this guidance is therefore to:

(i) embed best practice evaluation within our approach to strategic commercial investments and

(ii) provide an easily digestible summary of the key guidance on the best practice approach to evaluation as set out in HMT’s Magenta Book and Green Book, as well as the SPFM.

When should it be used?

As this guidance aims to highlight and summarise existing evaluation best practice, the key principles are applicable across SG and the wider Public Sector, however it is specifically tailored towards evaluating and learning lessons from previous and existing commercial interventions.

The guidance is intended to be a useful aid memoire for officials considering undertaking an evaluation of an intervention post-implementation. As a general rule, process evaluations should be undertaken six months to one year after the intervention, and outcome evaluations are best carried out after three to five years, or at a key decision point. Although, due to the variable nature of business interventions there will be an element of judgement as to when is a good time for evaluation to ensure the results of the evaluation are as accurate as possible and not skewed for instance due to a restructuring process.

As part of the business case undertaken prior to the intervention, a monitoring and evaluation plan should be developed to outline how monitoring and evaluation will be undertaken following implementation.

Useful Resources

HMT Magenta Book

HMT Green Book Scottish Public Finance Manual

Accountable Officer (AO) Tests

Scottish Enterprise - Appraisal and Evaluation Guidance (scottish-enterprise.com)