Strategic commercial interventions: evaluation guidance

5. Key considerations

5.1. Proportionality

The scale of any evaluation should reflect the scale of the investment in the business. High investment interventions should therefore undergo a more rigorous evaluation than smaller scale interventions.

Timing and the need for decisions to be made quickly may also have a bearing on the scale of the evaluation that can practically be undertaken in the time available.

5.2. Timing

Decisions on the timing of when evaluations should take place are largely subjective. This section sets out key issues to consider when determining the timing of each type of evaluation and suggests broad timescales.

• Process evaluation (lessons learned): The timing of a process evaluation has to be judged carefully to ensure that enough evidence is available to assess how well the intervention has been implemented but also to ensure that that most of the key individuals involved in delivering the project are still available to provide their input and that all issues encountered can be easily recalled and recorded. To ensure a balance between the two, it is considered that Process Evaluation should take place approximately six months to one year after the intervention.
• Outcome evaluation: An outcome evaluation should be conducted a sufficient period of time after the intervention to enable a thorough examination to be undertaken of actual outcomes against expected outcomes. If undertaken too soon, full impacts may not have had time to “work through”, but if undertaken too late, evaluation results may lose relevance. To ensure a balance between the two, it is considered that a full outcome evaluation should take place approximately three to five years after the intervention. Alternatively, a smaller scale outcome evaluation might be necessary at a key decision point, where information on the performance of the intervention is required to make a decision about the future of the intervention. For example, if Ministers need to decide whether to provide further financial support to the business, withdraw support etc.

5.3. Resourcing

Consideration should be given on a case-by-case basis as to whether the various parts of the evaluation can and should be undertaken in-house or commissioned externally. This is not just a consideration of whether we have the capacity to undertake the evaluation in-house, but more importantly, whether undertaking it in-house would ensure and demonstrate the independence necessary to assess these types of interventions rigorously. The availability of financial resources for the evaluation and the scale of the financial intervention (which should inform the scale of the evaluation) will also be key considerations.

• Outcome evaluation: Given the value and profile of many of these types of interventions, it is recommended that the outcome evaluation is commissioned externally, potentially with some input provided by Scottish Government Finance (SG Finance) and the Office of the Chief Economic Adviser (OCEA). The cost of commissioning externally would likely be proportionally small compared to the cost of the interventions themselves and would likely provide good value for money in supporting fully informed decisions going forward.
• Process evaluation: This may be undertaken internally or externally. Commissioning externally would likely allow for staff and stakeholders (particularly those outwith SG) to speak more freely and add independence.

To ensure consistency in approach across all strategic commercial asset evaluations, it is recommended that a steering group consisting of OCEA, SG Finance and Strategic Commercial Asset Division colleagues is set up alongside policy leads on each particular evaluation.

5.4. Determining attribution

The crux of an outcome evaluation is determining the ‘additionality’ of the intervention - to what extent any change in the outcomes monitored are a result of the intervention, as opposed to other external factors. This can be determined by comparing actual outcomes with those of the ‘counterfactual’, which is what is likely to have happened in the absence of the intervention.

Establishing the counterfactual can be challenging and requires judgment. The original business case undertaken prior to intervening should set out the ‘do nothing’ option and provide estimates of this which should form the basis of determining the counterfactual.

Where appropriate, consideration should be given to whether, in the absence of intervention, the business’ assets would have alternative uses and what capital expenditure would be required to create alternative uses, if this information is available. Consideration should be given to the likely impact on the local economy of not intervening in terms of the potential loss of the jobs and GVA supported by the business directly and indirectly through supply chain and re-spending of wages effects (for example, using input-output modelling), including the likelihood that any jobs lost will be absorbed into other parts of the economy.

5.5. Assessing geographic and distributional impacts

It is important to consider the geographical or other focus of the outcome evaluation. This should be informed by the objectives of the intervention, as set out in the business case. For example, if an intervention includes an objective to protect jobs in a fragile local area or for a particular group in society, it might be necessary to assess impacts at the overall Scotland level but also at the local level or for particular groups in society. This is particularly important where an intervention may ‘displace’ jobs and output from one local area or group to another so that there is limited or no impact at the overall national level, but more significant impact at the local level.