Independent Report - Evaluability Assessment - WPI Economics - Evaluation of Curriculum Reform Programme

Economic evaluation

Overview

Conducting an economic evaluation would allow the Scottish Government to understand if the education reforms provide value for money (VfM). The economic evaluability assessment considered the different economic evaluation approaches and where each approach might be appropriate, including Cost-Benefit Analysis (CBA), Cost-Effectiveness Analysis (CEA), Break-even analysis, 4Es VfM and Investment Evaluation.

Economic evaluations are typically a core part of the evaluation of large-scale government reforms because government departments are required to demonstrate that public money is being used efficiently and effectively and to demonstrate value for money. The choice of evaluation depends on the evaluation aims and objectives, data availability, and resources available for the evaluation (time and costs). Other evaluation strands planned as part of the evaluation strategy can help to facilitate an economic evaluation, for example, by collecting Management Information (MI) and cost information as part of a process evaluation or evidence of the impacts of the reforms to support the assessments of ‘benefits’ in a Cost-Benefit Analysis. Therefore, an economic evaluation should be planned alongside the delivery of other supportive evaluations rather than in isolation.

Evaluation aims and questions

As set out in the Magenta Book, an impact evaluation seeks to understand what difference an intervention made and will, where feasible, quantify the outcomes, but it cannot on its own assess whether those costs were justified. VfM evaluation considers this by assessing whether the intervention was a good use of resources or, in other words, whether the intervention was worth the investment. VfM evaluations often examine whether the benefits justified the costs, i.e. whether the benefits realised from the intervention were worth the resources invested. VfM evaluation can therefore improve the usefulness of the evaluation by focusing on how well resources are used, the value created and the justification of financial and opportunity costs.^[10]

The research questions for a VfM evaluation will depend on the type of intervention and the chosen evaluation approach. General research questions include:

• to what extent did the reform programme deliver value for money through good use of resources, bearing in mind its impacts and its costs?
• what was the value for money of the intervention?
  • what categories of costs and benefits are relevant to the project / programme?
  • which organisations bear the costs and what was funding spent on?
  • which organisations / entities can benefit?
  • can these costs and benefits be monetised? If so, how can they be valued?
  • do the benefits outweigh the costs?
  • what outcomes would have to change for the benefits to outweigh the costs?
  • what is the ratio of costs to benefits?
• is the intervention the best use of resources?
  • how does the ratio of costs to benefits compare to that of alternative interventions?
• how cost-effective was the intervention?
  • what was the cost per unit (outcome, participant, etc.)?

The intervention objectives, Theories of Change and Impact Evaluation approach will guide how the general research questions are turned into specific VfM research questions for the Education Reform Programme and CQAR. Below is a list of potential Education Reform Programme specific VfM research questions. This is not an exhaustive nor a prioritised list but is included to give a flavour of potential focus areas of a future VfM evaluation:

• what trade-offs were made in resource allocation (e.g. infrastructure vs. curriculum) and were they justified by impact?
• to what extent has the reform improved the efficiency of national education bodies by reducing operational costs while maintaining or enhancing service quality with no detriment to learners?
• did the sequencing of reform activities (policy • training • infrastructure) enhance or limit value for money?
• did the education reform process deliver outputs (Qualifications Scotland, HMIE, CfTE, single view on data and intelligence, curriculum updates to CfE through the CIC, etc.) on schedule and within budget?
• what contextual or governance factors helped or hindered VfM in this reform?

Economic evaluation approaches

There are a range of different types of VfM evaluation approaches and tools. When choosing the approach, it is important to consider the evaluation aims and objectives, the impact evaluation approach being undertaken, data availability, and resources available for the evaluation (time and costs). The evaluation aims and objectives will guide the research questions and different VfM tools can answer different research questions. With a complex intervention like the Education Reform Programme and CQAR, it is recommended to take a mixed methods approach using both qualitative and quantitative analyses, as one approach is not likely to cover the full range of areas of interest. Synthesis methods entail combining findings from a range of studies into a common understanding of the impact and/or delivery of a policy.^[11]

Below is an overview of different VfM approaches followed by a high-level assessment of which of the approaches might be suitable for the Education Reform Programme and CQAR project. Common to all of the approaches is that when conducting VfM analysis and analysis of costs and benefits of an intervention or policy, only the additional costs and benefits should be considered. The analysis will therefore have to consider which costs and benefits occurred as a result of the intervention and which would have occurred anyway. The counterfactual for the VfM evaluation should be aligned to the counterfactual from the impact evaluation. The impact evaluation measures the outcomes and impacts that are produced by the intervention. Adopting a QED approach to the impact evaluation can therefore support the VfM evaluation in quantifying the impacts, whereas a TBE would not do this to the same extent. As set out below, the counterfactual can then be proxied through other means or by using scenario analysis.

Cost-Benefit Analysis (CBA)

CBA or Social cost-benefit analysis compares the benefits of the intervention to the costs of the intervention. It is often what is thought of first when considering VfM. The approach involves collecting costs and benefits data via the monitoring and evaluation strands and quantifying and monetising these. The costs and benefits can be compared by calculating the present value of costs and benefits. They can also be combined into a benefit-cost ratio (BCR). The BCR measures the ratio of the present value of benefits to the present value of costs. It assesses the benefits bought about per pound of public spending.

The CBA approach is suitable when costs and benefits can be identified, quantified, and compared against a counterfactual scenario – often the Business as Usual (BAU) scenario and where the Theory of Change and M&E framework identify measurable and monetiseable outputs and outcomes. The approach is useful for capturing short- and long-term impacts systematically. It is one aspect of VfM and can assess the monetary value of impacts relative to costs. The quality of results relies heavily on available data and the ability to monetise the impacts. The CBA does not include other aspects such as intangible value, success factors (such as whether the strategic objectives were achieved), relevance, sustainability, affordability, acceptability, and others.

In the absence of a quantitative impact evaluation such as experimental, quasi-experimental or synthesis methods, a full CBA is not feasible as the impacts of the intervention cannot be quantified. This is likely to be the case for the Education Reform Programme, as discussed above. However, in some circumstances, the impact of the intervention or the additionality can be proxied using estimates from the literature for similar interventions and corroborated by the findings from a theory-based method. A CBA approach can then be used to illustrate whether the benefits outweighed the costs and whether the impact is likely to be of a size consistent with breaking even. This would often be supplemented by scenario and sensitivity analysis to account for the uncertainty of the impact findings.

Cost-Effectiveness Analysis (CEA)

CEA compares the costs of alternative ways of producing the same or similar outputs or outcomes. It explores how much it costs to achieve one unit of impact such as learners graduated, test scores improved, parent engagement scores, etc. CEA is generally used to compare options that aim to achieve the same type of outcome and enables comparisons between interventions (or intervention and do-nothing) where benefits cannot be converted into monetary units. It requires good quality data on both costs and outcomes.

CEA can be successfully applied where there are established, standardised comparable measures. For example, if evaluating two reading programmes in schools, CEA could compare the cost per student for an improvement in reading score (pound per reading point improved) for the two programmes. CEA does not account for wider social costs or benefits and so should not be used where different interventions might target different groups and outcomes. It can therefore be difficult to find a suitable comparator intervention. In the context of a complex reform, like the Education Reform Programme, it can also be challenging to obtain sufficiently detailed costs information and link costs to specific outputs and outcomes.

Break-even analysis

Break-even analysis can be applied to understand at what point the level of investment is returned, i.e. at what point do the positive outcomes outweigh the costs. For outcomes that can be quantified and monetised, the analysis assesses what level the outcomes would need to be at for the pounds/monetary benefits they bring to offset the costs of the intervention. Break-even analysis can be used when the costs of the intervention are known and there is a means of monetising (putting a pounds value) on the outcomes but is not possible (or appropriate) to estimate the impact of the intervention.

4Es Value for Money Evaluation

The National Audit Office’s (NAO) analytical framework for examining VfM – known as the “3 Es” of Economy, Efficiency and Effectiveness - or 4 Es when including Equity - is another way of assessing the VfM of an intervention. The proposed assessment of VfM will involve examining each of the four elements of the framework, identifying the links between them and drawing conclusions based on evidence about how well they perform together. If adequate data is collected, the 4Es approach can be used to track and monitor different outputs and outcomes in a programme and linking these to cost. The 4Es approach is typically recommended when it is difficult to quantify the benefits of an intervention through a CBA. The 4Es, which map on to the programme’s Theory of Change, describing the relationship between inputs, outputs, outcomes, and impacts are:

• Economy – this includes the costs of, and resources used to implement the intervention to achieve its intended outcomes. Is the programme getting the inputs (e.g. staffing, resources, materials) for the appropriate quality at the right price?
• Efficiency – how well are inputs (e.g. breakdown of costs) converted into outputs (“Spending well”)?
• Effectiveness – how well are the outputs produced by the programme having the intended impact (“Spending wisely”)? Here the impact evaluation can help link impacts to outputs.
• Equity – how fairly are the benefits distributed? To what extent are underrepresented groups reached? (“Spending fairly”)

The 4Es will be analysed in turn using data collected through the process and impact evaluation and, where applicable, secondary data sources. The robustness of the underlying process and impact evaluation will then determine the level of confidence in the VfM assessments made.

The 4Es framework is a structured way of assessing the VfM and combines different analyses to provide an overall view of VfM. The analyses can include cost breakdown, financial data, procurement processes (Economy), cost efficiency, monitoring data on outputs, interviews (Efficiency), impact evaluation findings, CEA, CBA (Effectiveness) and distributional analysis (Equity) and others. The approach does not produce a single measure to quantify the VfM, but rather assesses across the “Es,” sometimes multiple assessments for each “E” if appropriate.

4Es Value for Money/Investment Evaluation (Julian King/OPM framework)

This VfM assessment approach builds on the UK Department for International Development’s (DFID) Approach to VfM as set out above,^[12] which was further developed by Julian King, a VfM expert based in New Zealand and Oxford Policy Management.^[13] Like the National Audit Office framework, it assesses the VfM against the 4Es of Economy, Efficiency, Effectiveness and Equity. However, the 4Es Julian King/OPM approach also involves making transparent, evidence-based judgements about how well resources are being used, and whether the value derived is good enough to justify the investment. In so doing, the assessment makes use of ‘rubrics’ of criteria and standards. The Theory of Change is used to map input, outputs, and outcomes to the 4Es high level criteria of Economy, Efficiency, Effectiveness and Equity. Sub-criteria are identified which are the dimensions of performance that are relevant to the intervention. They are selected based on their ability to cover the most important aspects of VfM for intervention. After this, standards are defined.

The standards provide defined levels of VfM (i.e. excellent, good, adequate, and poor) for each of the criteria. The criteria and standards are then combined into rubrics. The rubrics thus represent an agreed statement about what matters and what good looks like. Involvement with key stakeholders is important to ensure that the agreed rubrics reflect the values intended, and to provide stakeholders with a clear understanding on which basis the VfM judgement will be made. This responds to criticism that the standard definitions of each ‘E’ can be too generic, and this approach benefits from using intervention-specific definitions of ‘economy,’ ‘efficiency,’ ‘effectiveness’ and ‘equity’ as well as having clear pre-specified standards of performance. Using the rubrics makes the VfM judgement on whether the intervention is ‘good’ or ‘excellent’ clear and transparent. The approach is particularly helpful where outcomes and impacts are not easily quantified and where the intervention is complex with a range out outcomes. The approach requires significant stakeholder engagement and comprehensive analyses of both qualitative and quantitative evidence and therefore might come at a greater cost than the NAO framework approach to 4Es.

Inclusion of underrepresented groups

Economic evaluation is less applicable to understanding the experiences and outcomes of underrepresented groups than impact and process evaluation. However, the 4Es approach considers equity as one of the dimensions against which the reforms are reviewed. This would allow for explicit consideration of the impacts across groups, including for particularly disadvantaged groups, and this may add to the case for this being a more suitable method.

Overall economic evaluability assessment

There are several ways economic evaluation could be applied to the Education Reform Programme and careful consideration would have to be taken to determine the best approach to this based on the methods set out above, as explained further below. Doing so would build on the impact- and process evaluation approaches put forward above, in line with best practice for major governmental reforms.

In addition, there are some aspects of the reform outcomes that could warrant specific economic evaluation. One example is the ‘efficiency outcomes’ which intend for the national education bodies to deliver improved organisational outcomes at reduced cost, without detriment to learners. It could be possible to explore the VfM specifically of the new or refocused bodies. For this, detailed information on cost and management information from the bodies themselves would be required as well as a good understanding of the outcomes and impacts.

The impact evaluability assessment highlighted that the reform programme is operating in a complex landscape and that it will be almost impossible to quantify and control for outside factors. It was also noted that many of the outcomes contained within the Education Reform Programme and CQAR Theories of Change can only be proxied by second best data. Therefore, it was recommended that the impact evaluation is focussed on TBE methods. For similar reasons, it will be challenging to adopt purely quantitative VfM evaluation approaches such as CBA to comprehensively assess the VfM of the reform.

Challenges in undertaking VfM evaluation when TBE impact evaluation is used include that the benefits of the reforms are likely to be realised with a long lag and may not be observed in the course of the evaluation period. This means a CBA approach would potentially need to rely on predicted future benefits or proxy leading indicators. The data mapping showed that several outcomes do not have clear indicators as they are intangible and hard to measure, making them particularly difficult to value. And finally, as touched on above, using a TBE approach makes it challenging to quantify the additionality of the reform and therefore to size the benefits to include in the CBA.

Therefore, it is recommended to use the 4Es VfM framework. This approach is also recommended by the UK Government’s Evaluation Taskforce,^[14] where intervention benefits are difficult to quantify. Both the NAO framework and the Julian King/OPM approach take a mixed methods approach using both qualitative and quantitative analyses and are therefore both comprehensive. The choice between the two will be a matter of the preference of the Scottish Government (or could be left for the evaluator to decide which they wish to use), the evaluation budget and the data collection strategy, which will be developed across process, impact and VfM evaluation.

Approach: CBA

Description: CBA or Social cost-benefit analysis compare the benefits of the intervention to the costs of the intervention and thus allows for transparency and accountability by setting out whether the intervention was worthwhile.

Challenges and limitations: The approach involves collecting costs and benefits data and quantifying and monetising these. In the absence of a quantitative impact evaluation, a full CBA is not feasible as the impacts of the intervention cannot be quantified.

Assessment: The data mapping did not identify indicators for many of the outcomes contained within the Education Reform Programme and CQAR Theories of Change and many outcomes can only be proxied by second best data. Therefore, it is recommended that the impact evaluation is focussed on TBE methods. For similar reasons, it will be challenging to adopt purely quantitative VfM evaluation approaches such as CBA to comprehensively assess the VfM of the reform. A partial CBA approach could potentially complement a 4Es VfM approach if it is possible to quantify and monetise some outcomes/benefits and the costs of the reform but would need to be caveated with sensitivity and scenario analyses. The Scottish Government and/or a future evaluator would need to assess the added value that could come from doing so, in relation to the additional resource required to undertake the analysis.

Approach: CEA

Description: Cost-effectiveness analysis compares the costs of alternative ways of producing the same or similar outputs or outcomes. It explores how much it costs to achieve one unit of impact such as students graduated, test scores improved, parent engagement scores, etc.

Challenges and limitations: CEA is generally used to compare options that aim to achieve the same type of outcome and enables comparisons between interventions (or intervention and do-nothing) where benefits cannot be converted into monetary units. It requires good quality data on both costs and outcomes.

Assessment: CEA can be successfully applied where there are established, standardised comparable measures. In the context of a complex reform, like the Education Reform Programme, it will be difficult to obtain sufficiently detailed costs information and link the costs to specific outputs and outcomes. It is therefore not recommended to focus on CEA as the main economic evaluation approach. There might be some aspects of the reform which can be partially assessed using CEA. For example, if there were clear pre-reform benchmarks of the costs of educational bodies and the outcome/impact was evaluated, the post-reform costs of the bodies could be compared to the pre-reform costs. The Scottish Government and/or a future evaluator would need to assess the added value that could come from doing so, in relation to the additional resource required to undertake the analysis.

Approach: Break-even

Description: Break-even analysis can be applied to understand at what point the level of investment is returned, i.e. at what point does the positive outcomes outweigh the costs. For outcomes that can be quantified and monetised, the analysis assesses what level would the outcomes need to be at for the investment to be deemed worthwhile.

Challenges and limitations: Break-even analysis can be used when the costs of the intervention is known and the value of the outcomes that are realised are also known. However, there is no estimate of the impact of the intervention.

Assessment: As with the CBA, break-even analysis could be applied to complement a 4Es evaluation approach. In particular, where it is possible to quantify and monetise some outcomes/benefits and where costs are collected as part of the monitoring data. The Scottish Government and/or a future evaluator would need to assess the added value that could come from doing so, in relation to the additional resource required to undertake the analysis.

Approach: 4Es – NAO

Description: The 4Es framework is a structured way of assessing the VfM. The analyses can include cost breakdown, financial data, procurement processes (Economy), cost efficiency, monitoring data on outputs, interviews (Efficiency), impact evaluation findings, CEA, CBA (Effectiveness) and distributional analysis (Equity).

Challenges and limitations: The approach does not produce a single measure to quantify the VfM but rather combines different analyses to provide an overall view of VfM. The analyses will need to be synthesised to provide an overall view of the VfM of the intervention.

Assessment: Using a theory-based impact evaluation approach makes it challenging to quantify the additionality of the reform and size the benefits to include in the CBA. Therefore, it is recommended to use the 4Es VfM framework. This approach is also recommended by the UK Government’s Evaluation Taskforce. The equity section allows for a specific focus on underrepresented groups, which is an area of interest for the Education Reform Programme.

Approach: 4Es – Julian King/OPM

Description: Like the NAO framework, it assesses the VfM against the 4Es of Economy, Efficiency, Effectiveness and Equity, but the 4Es Julian King/OPM approach involves making transparent, evidence-based judgements about how well resources are being used. In so doing, the assessment makes use of ‘rubrics’ of criteria and standards.

Challenges and limitations: The approach involves significant stakeholder engagement to agree the criteria and standards, which can be time consuming and likely to be higher cost than the NAO framework. However, once the rubrics have been agreed, there is a clear basis on which the VfM judgement will be made.

Assessment: The Julian King/OPM 4Es approach can include all the elements from the NAO 4Es framework as well as providing a transparent way of making a judgement of the VfM. The approach works very well with a TBE impact evaluation approach. The choice between the two 4Es approaches will be a dependent on the preferences of the Scottish Government, the evaluation budget, and the data collection strategy.