Strategic commercial interventions: evaluation guidance

Purpose, introduction and background

• The purpose of this guidance is to provide an easily digestible summary of the key guidance on the best practice approach to evaluation and make it relevant to strategic commercial investments.
• Evaluation is an assessment of the implementation and outcomes of an intervention. It takes place after the intervention and forms an essential part of the policy cycle, demonstrating what has been achieved with public resources and providing evidence and learning points for future interventions.
• It is essential that public funds are spent on activities that provide the greatest benefits to society and that they are spent in the most efficient way. It is therefore essential that strategic commercial investments are evaluated in a robust, consistent, evidence-based and objective-led way.

Outcome and process evaluation

• There are two main types of evaluation: outcome evaluation and process evaluation.
• Outcome evaluation provides an assessment of whether the intervention worked as expected – to what extent the expected outcomes were realised and at what cost. Specifically, it:
  • assesses whether the intervention met its objectives;
  • compares actual outcomes observed from the intervention with the outcomes expected for the counterfactual (which is what is likely to have happened in the absence of the intervention) to determine the extent to which the changes observed can be attributed to the intervention, as opposed to other external factors; and
  • provides an assessment of whether the intervention has provided value for money to date.
• Process evaluation provides an assessment of how well the intervention has been implemented and provides learning points to both improve the current intervention and to inform the implementation of future strategic commercial interventions. It may also be known as ‘lessons learned’. Specifically, it:
  • considers whether the intervention followed correct processes and procedures (audit-type check); and
  • draws on feedback from key internal and external stakeholders involved in the intervention to assess how well the intervention was implemented

Key things to bear in mind

• Proportionality: the scale of the evaluation should reflect the scale of the financial investment/intervention in the business. Timescales for decision making will also have an important bearing on the scale of the evaluation possible.
• Timing: broadly, process evaluations should be undertaken six months to one year after the intervention. Outcome evaluations should be undertaken three to five years after the intervention or, if necessary, at a key decision point.
• Resourcing: in considering whether the various parts of the evaluation can and should be undertaken in-house or commissioned externally, capacity and capability to undertake the work in-house as well as implications for the independence of the evaluation should be taken into account.