Independent Report - Evaluability Assessment - WPI Economics - Evaluation of Curriculum Reform Programme

Impact evaluation

Overview

The purpose of an impact evaluation is to explore the extent to which an intervention or reform has delivered the intended results. Conducting an impact evaluation will allow the Scottish Government to understand the changes which the Education Reform Programme brings about.

Impact evaluations can be applied to all types of outcomes, but for complex reforms, they often focus on priority outcomes and key causal pathways to make the evaluation more feasible. This will be important in the case of the Education Reform Programme, where the outcomes span several groups and parts of the education system, and arise over extended time periods as the reform activities are implemented and benefits are realised.

There are then a number of approaches that can be taken to impact evaluation, spanning those which draw heavily on quantitative data, to those which adopt more qualitative, theoretical approaches. Two primary methods of impact evaluation have been considered, which represent each side of this spectrum, Theory-Based Evaluation (TBE), and quasi-experimental design (QED). This consideration draws heavily on the wealth of knowledge contained on these methods in the Magenta Book^[1].

Evaluation aims and questions

An impact evaluation captures the measurable achievement of an intervention. This may feature measuring objectives of the intervention itself, or contributory elements of those objectives.

The overarching research questions for a future evaluator would typically be:

• what are the measurable outcomes of the Education Reform Programme?
• to what extent can these outcomes be attributed to the reforms?
• have different groups been impacted in different ways by the reforms?
  • how and why?
• how has the context influenced these outcomes?
• have the reforms contributed to any negative consequences or outcomes?

Given there are a large number of outcomes across the Education Reform Programme and CQAR, it is recommended a future evaluator determines a short-list to focus on, formulating a more specific set of research questions depending on the methodology adopted (this is explained further in the next sub-section).

In addition, before establishing the research questions, it would be beneficial for the Scottish Government to develop the high-level reform objectives into SMART objectives^[2] to provide clear, actionable and trackable goals that can improve performance and at the same time provide a clear focus for the evaluation. For example, for the high-level objective ‘Design and deliver a national organisational infrastructure for education in Scotland that more effectively supports the system as a whole to deliver the vision for education in Scotland’, a SMART objective could be ‘By [December 2031], design and implement a national organisational infrastructure for education in Scotland that provides integrated support services, governance structures, and coordination mechanisms, with at least a 20% reduction in the costs of services provided by the national education bodies with no detriment to learners while delivering the national vision for education.’

For a Theory-Based Evaluation (TBE) approach, in which measuring outcome achievement does not depend on the existence of quantitative data, an evaluator would short-list outcomes to analyse based on the development of detailed causal claims. These claims will focus on testing specific linkages of activities through to outputs and subsequently outcomes.

For quasi-experimental design (QED) approaches, outcomes which can be evidenced will require appropriate underpinning quantitative data. The data mapping exercise carried out prior to this assessment contains judgements as to which outputs and outcomes can be robustly evidenced given existing data sources. Combining this data with what might be collected as primary data in the evaluation will determine the short-list of outcomes which would be the focus of the impact research questions, if adopting a QED approach.

Impact evaluation approaches

Impact evaluation approaches broadly fall into two categories:

1. Theory-Based Evaluation (TBE)

Theory-based impact evaluations draw conclusions about an intervention’s impact through rigorous testing of whether the causal chains thought to bring about the change are supported by sufficiently strong evidence. Evidence is also used to rule out alternative explanations.

TBE can help to establish both (a) the extent of the change and (b) why change occurs, considering the linkage between inputs and outcomes, and how that is affected by wider contexts. It is not necessarily limited to assessing quantitative outcomes but can also make use of qualitative analysis to demonstrate casual links between the reforms and the intended benefits.

Commonly used approaches include:

• contribution analysis, a step-by-step process used to examine if an intervention has contributed to an observed outcome, by exploring a range of evidence for the Theory of Change outcomes, and giving an evidenced line of reasoning, rather than a definite proof
• Bayesian updating estimates and tests the probabilities of contribution claims prior to observation, and can be added to other TBE approaches to more rigorously assess whether evidence supports contribution claims
• contribution tracing is a participatory mixed-method approach which establishes the validity of contribution claims, consolidating data collection and Bayesian updating to quantify the level of confidence in a claim

Other forms of TBE include realist evaluation, process tracing, qualitative comparative analysis, outcome harvesting and most-significant change. A future evaluator would consider the best specific methodology or a hybrid approach as part of the detailed evaluation design.

The Education Reform Programme, and context surrounding it, complies with several criteria which point towards TBE being an appropriate impact evaluation method in line with the Magenta Book guidance. The reforms:

• sit in a dynamic environment
• have unclear or soft outcomes
• have complex causal pathways
• may vary in implementation
• are implemented in a context which is likely to matter
• have a medium to long-term timescale

However, whilst these approaches all allow attribution of causality, none give precise estimates of effect sizes. Furthermore, some quantitative analysis could be helpful to supplement the findings of a TBE and monitor how the programme is performing on key indicators identified in the data mapping stage.

2. Experimental / QED approaches

Experimental and QED approaches use quantitative methods and statistical techniques to enable comparison of treatment (those who are subject to the intervention) versus control groups (those who are not subject to the intervention).

The core principle of trying to create the control / counterfactual group is that it should have effectively identical characteristics to the treatment group. This is so that the status of whether the intervention has applied to the individuals is the only difference between the groups, thus providing evidence for the impact of the interventions if there are observed differences in outcomes between the groups. Where there are certain commonalities amongst individuals in the treatment group, which are not found in the control group, a causal impact of the intervention cannot be established because the other common factor(s) could be driving the outcome. Experimental and QED approaches address this issue by either randomising who receives the intervention in an experimental design, or by making appropriate analytical adjustments (in QEDs).

Experimental approaches

Randomised Control Trials (RCTs) are considered the gold standard for causal inference and must be incorporated in the design phases of an intervention. For this method, participants are randomly allocated to intervention or control groups. The randomisation means that measured differences between groups can be attributable to the intervention alone. Randomisation must be carefully considered as part of an intervention’s design, given that it may not be a feasible and ethical method of delivery.

Given that the Education Reform Programme is already underway, randomisation of treatment is not possible. All stakeholder groups in Scotland-wide will be influenced by the reforms, without discrimination for which learners / parents / stakeholders will be affected by them.

The design of an RCT in this context could also raise practical and ethical challenges, in terms of limiting some children’s education experience. An RCT can therefore be ruled out.

QED

QED approaches create comparable treatment and control groups by using alternate methods to randomisation. Some natural experiments can create conditions which analysts can use to mimic randomisation, for example, there might be external factors which influence whether an individual is affected by an intervention which are unrelated to the outcomes of interest (the instrumental variable approach relies on this). Some commonly used methods include:

Difference in difference (DiD) methods use time-series data to test for a causal change in the trend of outcomes following an intervention. This relies on comparison of trends for a treatment group to a control group, which was experiencing ‘parallel’ trends prior to the intervention. Hence, the DiD method assumes that the treatment group would have been on the same trajectory as the control group, in the absence of the intervention.

Regression Discontinuity Design (RDD) constructs control and treatment groups if eligibility for treatment depends on an arbitrary cut-off or threshold (such as age or date). Given that individuals just above and below the threshold can be considered as essentially very similar (it is in essence chance who falls on either side of the threshold which is set), who is in the treatment or control group effectively mimics randomisation in this instance.

Instrumental variables rely on identifying a variable which influences whether or not treatment is received, but crucially this variable has no connection to the outcome variable being measured.

Interrupted time series compares the trends of a dependent variable before and after the onset of an intervention and attempts to detect and/or measure either immediate or long-term changes in the dependent variable’s trajectory over time. This method is particularly relevant to cases where no control group, synthetic or otherwise, can be constructed. Using interrupted time series for inference requires the assumption that key background variables hold constant both before and after the intervention takes place and also relies on being able to determine the time at which changes due to reforms should feed through into outcomes in the dependent variable.

Other QED methods based on similar principles include creating synthetic controls and propensity score matching. Again, a future evaluator would consider the best specific methodology or a hybrid approach as part of the detailed evaluation design.

QED methods would allow quantifiable impacts of the education reforms to be estimated. Crucially, their robustness relies on the availability of high-quality data sources, and well-constructed methodologies which conform to relevant statistical tests (e.g. the ‘parallel trends’ assumption in the DiD method).

QED has most promise with respect to evaluating outcomes that can be proxied or partially proxied using existing datasets. Outcomes of this kind identified in the Theories of Change set out in our previous report include:

Education Reform Programme

A forward-facing curriculum, supporting learners’ progression – Scottish Government and Skills Development Scotland data regarding pupil destinations and planned vs actual destinations could be particularly relevant here.

A credible, equitable, accessible, valid and reliable qualifications and assessment system – some aspects of this outcome could be measured using the Achievement of Curriculum of Excellence Levels data, which provides demographic breakdowns that could allow for equity in outcomes across groups to be measured over time.

CQAR

Programme of professional learning to support the roll-out and implementation of the improved framework – international assessments like the Trends in International Mathematics and Science Study (TIMMS) and Progress in International Reading Literacy Study (PIRLS) include teacher questionnaires including questions regarding recent uptake of training.

Learners [are] able to achieve full potential through positive attainment at the chosen point of exit – the Scottish Government records a wealth of data regarding learner academic attainment (though the Achievement of Curriculum of Excellence levels data) and subsequent destinations, with snapshot data both at three and nine months after leaving education, and for all 16 to 19 year olds in Scotland through the Skills Development Scotland’s Annual Participation Measure. Skills Development Scotland also record data regarding the planned destinations of senior phase learners and the difference between these plans and the actual destinations of the pupils.

However, a crucial reason QED approaches are not recommended for the Education Reform Programme evaluation are the difficulties in comparing individuals who are affected, or not affected, by the reforms. As previously mentioned, the reforms in Scotland are system-wide and without discrimination for which learners / parents / stakeholders will be affected by them (there is no way we can artificially randomise who receives treatment or not).

There could be some potential ways to construct comparator groups for Scotland going through the reforms, in a synthetic control setting potentially – for example, by considering trends which have been experienced in comparable geographies. However, there could be difficulty finding countries with similar circumstances and trajectories to Scotland, given different countries will have been enacting their own education reforms at varying times.

Ultimately, the reform programmes are operating in a very complex landscape, and it would be difficult to attribute direct impact in such environments. A paper by the IFS for the World Bank and the Thematic Group on Poverty Analysis, Monitoring and Impact Evaluation on the use of impact evaluation for school-based reforms comments that where there is nationwide coverage of school reforms, whilst it might be possible to conduct before and after comparison using an interrupted time series design, the major drawback is that these aggregate trends in the outcome variable fail to isolate what has been driven by the reforms themselves, and it is impossible to control for all factors affecting the outcome.

Further, as established in the data mapping phase, many of the outcomes contained within the Education Reform Programme and CQAR Theories of Change can only be proxied by second best data. Hence, even if a credible QED approach could be established to measure these proxies, it would only go part of the way to understanding the achievement of a wider outcome or objective.

Ultimately, while some form of quantitative analysis could be helpful to strengthen the credibility of the TBE, and a pre-post analysis of key indicators will be important to track in the evaluation, it is WPI Economics’ recommendation that experimental / QED approaches should not be the sole / principal means of impact assessment for the Education Reform Programme. It is instead recommended that the impact evaluation is focussed on TBE methods.

Data collection

Some examples have been provided above of where existing secondary data sources could be drawn in as part of the evidence to assess impact within a TBE framework, even if not as part of a QED approach – for example, using this data to monitor general trends in key education metrics whilst the reforms are taking place.

Within a TBE approach, it is also likely to be necessary and appropriate that a future evaluator conducts (proportionate) primary data collection to allow the impact evaluation to go as far as possible in determining whether the desired outcomes of the reforms have been achieved.

Qualitative data collection (in the form of workshops, key stakeholder interviews and focus groups) will be essential in corroborating or discounting the role of the education reform activities and outputs in driving longer-term outcomes. The stakeholders which are consulted will have implications for which causal chains within the Theory of Change are tested.

More quantitative evidence could then be gathered to supplement qualitative findings. Survey methods, for example, could be used to produce insights that are more bespoke to the education reforms. Where surveys are in the control of the Scottish Government or affiliated bodies, it could be that existing studies and questionnaires are tweaked to formulate bespoke findings to the Education Reform Programme. Alternatively, a future evaluator may look to commission a bespoke survey, which would need to balance appropriate response rates (and hence sample sizes) with resource constraints.

Inclusion of underrepresented groups

There are a number of underrepresented groups that have specific needs and/or currently experience differential outcomes in the education system. This includes learners with additional support needs, socioeconomic disadvantage, care experienced learners, home schooled learners, and Gaelic medium education (GME) learners and educators.

There are two key ways in which impact evaluation approaches are relevant to consider the outcomes of the reforms for these groups. Firstly, to evaluate whether the reform activities that are intended to improve the equity and accessibility of key parts of the education system have delivered their intended impact, and secondly to consider any differences in the outcomes of the reforms across groups. This will be relevant for both learners, and educators and parents/carers for whom that the Scottish Government intends to improve engagement and support for through the delivery of the reforms.

In principle, any impact evaluations undertaken can consider the outcomes for underrepresented groups, alongside the impact for all learners, educators, or stakeholders, by making use of data and primary research that captures and allows for analysis of the specific experiences of the groups of interest. This typically needs explicit consideration, as these groups can be less likely to engage with research and be poorly represented in terms of survey sample sizes. Quota sampling can be used to increase representation of these groups for quantitative data collection. As set out in the Magenta Book, there are a number of sampling methods that could be used, but two approaches that have aided previous evaluations by improving responses from underrepresented groups are:

Respondent-driven sampling (RDS) – a type of snowballing sampling method, which begins with a small sample of initial participants (called “seeds”) who are provided incentives for completing a survey or interview, and another for successfully recruiting additional participants. Statistical adjustments are then applied to reduce biases within the data collection, primarily on the basis of network size and network characteristics. RDS is a time-intensive process, dependent upon well-connected initial participants with multiple waves of peer-to-peer recruitment large enough to generate a good sample without the risk of recruiting repeat participants.

Time-location sampling (TLS) – a probability-based method that samples participants at specific times and locations where participants from underrepresented groups are known to meet. This requires identifying all possible locations, understanding time slots, randomly selecting venues from the long list of locations, and recruiting either a random or systematic sample at these selected locations during the known time slots. Weighting is then applied based on how large or busy the locations are by the inverse of their selection probability (i.e. how likely an individual is to be at a specific location at a specific time).

The data collected via these approaches could also be used qualitatively provided there are the important caveats noted around.

In addition, speaking to representative bodies for these groups is critical. This can serve the purpose of both engaging their perspectives as well as utilising these organisations as important gatekeepers and potential peer researchers.

Another option is that the Scottish Government may wish to prioritise understanding the impact of the reforms for underrepresented groups and commission an impact evaluation specifically for this purpose, or to focus on the evaluation of the impact of the outcomes that relate to improving equity and accessibility in the education system. This would likely allow for greater depth of primary research and data collection to understand the outcomes of the reforms for underrepresented groups, as well as to understand the casual links between the reform activities and the outcomes identified, and the reasons for any differential impacts or experiences.

Summary of impact evaluation approaches

This section has explored two categories of impact evaluation methods – TBE and QED – their appropriateness in this context, and the bespoke analytical approaches which could apply within these. Whilst TBE would mean adopting a more qualitative approach to testing the causal impact of the education reforms, QED methods rely on the use of robust data and methods to establish appropriate counterfactual scenarios against which the impacts of the reforms can be measured quantitatively.

The table below summarises the overarching assessment which has been made for the two impact evaluation approaches and the recommended approach to pursue for the Education Reform Programme. This recommends the use of TBE evaluation as a primary evaluation method for the monitoring and evaluation strategy, allowing for analysis of the impact of the reforms.

The innate challenge with impact evaluation is that in principle it can be used to measure the impact of all the outcomes of the reforms, and the timelines over which different benefits would be expected to be realised. Further consideration will need to be given as to how to apply this approach to the Education Reform Programme in practice, and the exact focus and timing of any impact evaluation(s) commissioned. This could include for example, separate evaluations of the intended outcomes for specific focus areas of the reforms such as the improvements in collaboration and leadership across the education-system, or changes to the curriculum, assessments, and qualifications system. Furthermore, an impact evaluation could be selected to focus specifically on whether the reforms improved engagement and accessibility within the education system, and the impact of the reforms for underrepresented groups.

Approach: Theory-based (TBE)

Description: These methods are used to draw conclusions about an intervention’s impact through the rigorous testing of whether the causal chains thought to bring about change are supported by sufficiently strong evidence.

Challenges and limitations: While this form of methodology can often draw on a significant basis of evidence, it relies on the causal chains correctly capturing all of the most important factors determining intervention outcomes, and the relationships between these factors.

The qualitative data collection required to drive these methods will also be time intensive, hence the resourcing implications could limit the scope of this method.

Assessment: These methods – as outlined above and in the Magenta Book – are likely to be most useful in cases where the effects of an intervention cannot be measured directly, as is the case for Education Reform Programme. We have determined that these methods are suitable to be the principle means of impact evaluation in this case.

Approach: Experimental / Quasi-experimental designs (QED)

Description: These methods use quantitative methods and statistical techniques to enable comparison of treatment versus control groups. They therefore allow us to directly measure the impact of an intervention on a dependent variable.

Challenges and limitations: While these methods are the most robust way to measure the impact of an intervention directly, they rely on the formulation of a control group, synthetic or otherwise, which can act as a counterfactual and thus allow us to measure the changes which are exclusively due to the intervention. In this case, because the intervention applies uniformly across the whole population, a valid control group will be difficult and potentially inappropriate to find or construct.

Assessment: As outlined above and in the Magenta Book, these methods are unlikely to be viable in a situation where effective control populations cannot be easy created, found, or constructed after the fact. As a result, we do not recommend these methods to be used as principle means of impact evaluation in this case, though it may be appropriate for some quantitative methods to feed into the impact assessment of some outcomes.