Independent Report - Evaluability Assessment - WPI Economics - Evaluation of Curriculum Reform Programme

Conclusions

Having completed the evaluability assessment, we would recommend that all three types of evaluation (impact, process and economic) can meaningfully be used as part of the M&E Strategy for the Education Reform Programme. Having reviewed the nature of the outcomes, likely data availability, and context of the reforms, WPI Economics has selected a recommended approach for each evaluation as detailed in the table below.

Evaluation type: Impact evaluation

Summary of recommended approach: After considering both theory-based and QED methodological approaches to this impact evaluation, we determined that while there may be some cases where QED can feed into the impact evaluation for some outcomes, TBE is preferred as the primary methodology for conducting the impact evaluation. This is due to likely limitations with respect to finding or constructing robust counterfactuals for the vast majority of outcomes, with the result that most QED methodologies will be either impossible to apply or likely to produce uninformative outputs. In contrast, TBE approaches are more viable across a much broader spectrum of outcomes.

Evaluation type: Process evaluation

Summary of recommended approach: Given the purpose of process evaluations, it is of most value to conduct them at an interim stage as well as at the end of the delivery. This assessment determines that a mixed methods approach should be used for interim stage evaluations as well as to evaluate specific aspects of the project such as inclusion of different underrepresented and stakeholder groups in the reform process. A process evaluation for the overall Education Reform Programme would be resource intensive and should only be considered if the Scottish Government is interested in extracting learnings from how the reforms were implemented and what could improve effectiveness for implementing future reform processes. Given the complex nature of the reforms and the number of distinct activities that feed into the reform programme and the resulting outputs, a Qualitative Comparative Analysis approach could be used. This would supplement a mixed-method process evaluation and an overall impact evaluation to gain a high-level and programme-wide understanding of what went well and identify any challenges.

Evaluation type: Economic evaluation

Summary of recommended approach: The assessment considered CBA, CEA, break-even and 4Es approaches to the economic evaluation. Since intervention benefits are difficult to quantify and the evaluability assessment is recommending a theory-based approach to impact evaluation, it is recommended to use the 4Es VfM framework. Both the NAO framework and the Julian King/OPM approach take a mixed-methods approach using both qualitative and quantitative analyses and are therefore both comprehensive. The choice between the two will be a matter of the preference of the Scottish Government, the evaluation budget, and the data collection strategy, which will be developed across process, impact, and VfM evaluation.

The next stage of work will be to identify evaluation priorities and parameters (such as cost and time), which should inform how these recommended evaluation methods are applied in practice and in a proportionate and targeted way. Given the timeline of the reforms, it is likely that evaluations should be carried out at different points. This will also be determined by the extent to which the Scottish Government wishes to prioritise early-stage evaluations so that lessons emerging during the reform process can be incorporated into the later stages of delivery.

In principle, it is possible to conduct an impact, process and economic evaluation using the methods set out above for the Education Reform Programme as a whole. However, there may be benefit in considering the use of smaller-scale targeted studies based on the specific areas of interest or clusters of work. This could be informed by the type of participants needed to take part in primary research as part of an evaluation to consider particular groups of outcomes, or when different parts of the reforms will be implemented.

An initial view of how each type of evaluation could be incorporated into an evaluation plan is set out below:

Impact evaluation

• an overall impact evaluation for the Education Reform Programme
• an impact evaluation of changes to the national educational infrastructure and system-wide collaboration to evaluate the extent to which collaboration, leadership and stakeholder engagement outcomes have been achieved
• an impact evaluation of the CQAR project, with particular focus on the outcomes for learners and underrepresented groups

Process evaluation

• an early-stage multi-method process evaluation of the curriculum reform process, from which learnings can be used within the next stages of the CQAR reforms
• a specific process evaluation(s) for the inclusion of underrepresented groups and stakeholders in the Education Reform Programme (and CQAR project)
• an overall process evaluation for the Education Reform Programme and CQAR upon programme completion, potentially making use of a QCA methodology to supplement the mixed-method process evaluation approach and overall impact evaluation

Economic evaluation

• alongside any impact evaluations, an economic evaluation utilising the 4Es approach to determine the overall VfM of the reforms or the national education infrastructure