Strategic commercial interventions: evaluation guidance

7. Process evaluation

Process evaluation provides an assessment of how well the intervention has been implemented and provides learning points to both improve the current intervention and to inform the implementation of future strategic commercial interventions.

Process evaluation should consider both objective questions around whether the intervention followed correct processes as well as more subjective questions to gather information on the perceptions of key stakeholders about the delivery of the intervention to provide the richer, qualitative information. For the latter, information can be collected through face-to-face interviews with those involved with the intervention - within SG, the business and any other relevant parties - or by more structured methods such as questionnaires. Interviews should be carried out in a constructive manner in order to encourage openness and honesty about what went well and what aspects could have been improved upon.

7.1. Objective questions

A detailed checklist is provided in Annex 1, however in particular the process evaluation should consider the following objective questions:

• Did the intervention align with key policy objectives at the time?
• Was a business case for the intervention developed in line with the HM Treasury Green Book five case model?
• Were SMART objectives for the intervention identified, set out and agreed?
• Were expected outcomes of the intervention set out?
• Was a clear exit strategy developed?
• Was sufficient time taken to consider the advice and options available including the option of not intervening?
• Was proper due diligence undertaken prior to the intervention?
• Were the correct approvals obtained prior to intervention?
• Was the independence of all individuals and entities involved in the transaction recorded and maintained?
• Was the intervention in line with SPFM guidance?
• Were proper project management processes put in place?
• Was a risk management register generated and updated throughout?
• Were suitable safeguards put in place to support SG’s interests in the intervention?
• Were proper record and data management processes in place?
• Was appropriate monitoring and reporting undertaken throughout the intervention on progress against expected outcomes, financial cost of the intervention, risk exposure etc?
• Have proper accounting terms been applied to the intervention (e.g. a contingent liability)?
• Was there a clear Ministerial structure in place to support the intervention?
• Were external advisors consulted where necessary?
• Were the correct policy teams engaged at the appropriate points?
• Was sufficient resource and the right skillset made available for the intervention?
• Was the correct SG process followed for the spend?
• Were SG spending controls adhered to? Was the approval matrix followed in line with expectation?
• Have Accountable Officer and Minister advice/approvals been recorded? (A link to the Accountable Officer (AO) Tests that require to be satisfied as part of the approval process is provided here).

This is not an exhaustive list of areas that should be examined, but a suggested baseline to be considered on a case-by-case basis. It may be useful to engage with SG Internal Audit in considering the above questions as they can provide expertise in ensuring internal process are followed and internal controls are operating effectively, particularly those around approval limits.

7.2. Subjective questions

As discussed above, in addition to the audit type checks outlined above, qualitative insights and feedback from key stakeholders (internal and external) involved in the delivery of the intervention about the delivery, handling and communications should also be assessed as part of a process evaluation to help learn lessons for future delivery. In particular, process evaluation should consider the following more subjective questions:

• What worked well / less well and why?
• What could be improved?
• Whether the type of intervention (e.g. loan/equity/guarantee) is working as expected and providing the support needed to the business or whether a different type of intervention might have been better?
• How has the context influenced achievement of objectives? (this should allow external factors, and the extent to which they have influenced the achievement of objectives, to be considered)