Strategic commercial interventions: evaluation guidance

6. Outcome evaluation

Outcome evaluation provides an assessment of whether the intervention has met its objectives and whether it remains value for money. It provides an assessment of what changes have occurred following the intervention and the extent to which those changes can be attributed to the intervention, by comparing with the counterfactual (what would likely have happened in the absence of intervention).

i. To assess the success of the project in achieving its objectives and the extent to which any changes that have occurred can be attributed to the intervention, actual outcomes on key indicators derived from the objectives of the intervention (as set out in the business case), quantified where possible, should be compared with expected outcomes and those of the counterfactual, as set out in the business case.

While external consultants would often be required to undertake this work, OCEA can undertake a narrow economic assessment of the likely economic impact of the intervention in terms of the number of jobs and GVA supported by the business both directly and indirectly through supply chain and re-spending of wages effects compared with what would likely have happened in the absence of the intervention. This would make use of:

• input-output analysis: this would require information on the number of full-time equivalent jobs directly employed by the business (excluding any contractors) in each site in each year since the intervention.
• a brief analysis of official statistics on the relevant sector and local authority area to determine the extent to which any jobs that may have been lost had the government not intervened would have been likely to have been absorbed into other parts of the local or national economy.^[2] To support this analysis, information on the make-up of the business’ workforce in terms of types of occupations, age profile and where the workforce is resident (e.g. in the local area or not) would be useful.

It is important to note that this would only form one narrow part of the overall assessment of the success of the intervention in meeting its objectives. External consultants may be required to conduct the broader work.

If an objective of the intervention was to return the business to commercial viability so that it could operate without government support in future, then an assessment of the business’ financial health and commercial viability post-intervention compared to pre-intervention should be included here. This would include:

• an examination of key financial indicators including profitability, cashflow, balance sheet and basic ratio analysis of the accounts, comparing these to similar businesses in the same industry
• an assessment of any key risks identified from the financial data
• an examination of the business’ order book, key customers/suppliers and their bargaining power, the type of contractual arrangements they may be exposed to (this is often industry specific)
• an assessment of the opportunities and challenges in the wider market/industry the businesses operate within and the state of the wider economy
• any change in ownership structure or changes to Directors
• obtaining audit opinions from the Annual Accounts and explaining their potential impact on the company

SG Finance may be able to assist with the financial aspect of the above (first two bullets) however it is likely that external consultants would be required to undertake the commercial viability aspect.

ii. The economic (social) costs and benefits of the intervention should be set out, quantified where possible, and discounted to ‘present value’.^[3] This should include both a revised assessment (based on actual outturn costs and benefits) of the expected costs and benefits of the intervention as set out in the business case as well as an assessment of any unexpected or unintended costs/benefits of the intervention that occurred. Here, we are considering impacts on those external to the business itself, e.g. the local economy, the users of the business’ assets or services, environment etc. The nature of many of the economic benefits of strategic commercial interventions may mean that quantifying them is difficult and it may therefore be necessary to describe them in qualitative, rather than quantitative terms. Economic costs should be easier to quantify than the benefits. The main difference between economic and financial costs is that economic costs should: distinguish between operating/running and capital costs where relevant; exclude ‘sunk’ costs; be discounted to covert future costs into their present value; and exclude inflation and VAT.

iii. To assess whether the intervention has provided value for money to the taxpayer to date, usually, a benefit cost ratio (BCR) would be calculated. However, given that many of the benefits associated with these types of interventions may be intangible and therefore non-quantifiable, it may not be possible to calculate a BCR. An alternative indicator of value for money is the total economic cost to government of the intervention: the sum of the discounted costs (operating and capital) minus the sum of the discounted revenue, excluding any capital receipts.

The evaluation should utilise information gathered for monitoring purposes during the intervention as well as existing secondary data where appropriate.