Independent Report - Evaluability Assessment - WPI Economics - Evaluation of Curriculum Reform Programme

Process evaluation

Overview

The purpose of a process evaluation is to assess how interventions or reforms have been implemented over the course of a project. Conducting a process evaluation will allow the Scottish Government to evaluate how the interventions of the Education Reform Programme have been implemented in practice and assess why they worked (or did not). It can provide greater insight into the implementation process and facilitate understanding of its effectiveness.

Process evaluations are usually focused on activities and, to a lesser extent, their outputs. The approaches which can be applied for these evaluations tend to focus on analysing qualitative data and the experiences of various stakeholders as well as quantitative administrative data. Process evaluations provide most value where the reform process is to be replicated or adapted, or learnings can be applied to similar interventions in the near future. As such, there are some specific aspects of the reforms that process evaluation could be most applicable to, for example, the Curriculum Improvement Cycle (CIC), as well as in considering how different stakeholders and underrepresented groups have been included and prioritised within the reform activities.

Evaluation aims and questions

Process evaluations tend to examine activities involved in an intervention’s implementation and the pathways by which the policy was delivered. These might vary quite considerably according to the nature of the intervention and will be policy specific.

The overarching research questions for a future evaluator include those that can draw insights from the delivery of the intervention as well as what can be taken into account for future projects:

• what can be learned from how the intervention was delivered?
  • are the interventions working as intended?
  • were there any unexpected or unintended issues in the delivery of the intervention?
  • to what extent has the intervention reached all the people that it was intended to?
  • what could be improved?
  • what can be learned from the delivery methods used?
  • how did external factors influence the delivery and functioning of interventions?
• what future learning is there from these reforms?
  • are the intervention’s goals relevant in different contexts (e.g., different geographies, different target groups etc.)?
  • can the policy be expected to work in other contexts?
  • is the intervention sustainable from financial, economic, social, and environmental perspectives?

Given that there are a substantive number of activities across the Education Reform Programme and CQAR, it is recommended that a future evaluator determines a short-list to focus on, formulating a more specific set of research questions including but not limited to, for example:

Education Reform Programme

• has the remit of Qualifications Scotland, HMIE and Education Scotland been defined?
  • is there now a system-wide understanding?
  • are ways of working defined and implemented?
  • what is the operational status of the bodies?
  • what worked and what didn’t in the process of operationalisation?
• has the Digital Vision and Strategy for national education bodies been agreed and implemented?
  • what existing solutions have been carried forward and what new ones have been introduced?
  • in each case, why?

CQAR Project

• has the evolved technical framework been successful in providing clarity on what learners should be able to know, do, and understand at each stage?
  • who are the intended target groups?
  • have all of these groups achieved clarity?
  • what areas of the curriculum are less clear?
  • for whom are they less clear?
• has a new programme of professional learning for educators been developed?
  • are there sufficient training resources in all the subject areas?
  • what are the gaps and for which stakeholder(s)?

Process evaluation approaches

Process evaluations typically use a wide range of methods, both quantitative and qualitative. They will often cover subjective issues (such as perceptions of how well a policy has operated) and objective issues (the factual details of how an intervention has operated, typically using administrative data, where available).

A mixed methods approach is often utilised to ensure a wide range of qualitative and quantitative data is captured and analysed. Some process evaluation methods under this approach include: ^[3]

• interviews to enable in-depth exploration of the intervention with participants
• focus groups to elicit views from a group of people rather than an individual
• case studies to conduct in-depth investigation of a person, group, or event within its real-world context
• surveys to collect data from a number of individuals, administered online or in-person
• output or performance monitoring for continuous measurement and performance review of an intervention
• observational studies (including ethnography) that involve observing and noting behaviour of participants (including intervention delivery staff)

This approach is especially useful for evaluating the interim stages of the project delivery process and to assess specific activities or outputs at various mid-stages of the project. Mixed methods can be used to gain deeper understanding at the individual level as well as at the level of a department, class, institution, or a specific stakeholder group. Limitations to this approach include it being resource intensive, having a risk of bias, and being difficult to generalise findings to different contexts.

Another potential approach that would help a future evaluator to evaluate the project as a whole is Qualitative Comparative Analysis (QCA). This could be useful in understanding in broad terms how the more successful reform elements varied from those that were less so. It may be useful to select some more detailed case studies based on an assessment of, for example, which ones are particularly successful, or which seem particularly innovative.^[4]

The Magenta Book states that the QCA method is used to compare multiple cases (activities and outputs, in this case) and systematically understand patterns of characteristics associated with desired or undesired outcomes based on qualitative knowledge. It can account for both complex causation (combinations of factors) and equifinality (multiple causes of outcomes). It can also identify groups of causal factors in post-hoc evaluation and can systematically analyse case study evidence. It works best with 10-50 cases, which corresponds to the number of interventions across the Education Reform Programme and the CQAR. One caveat to this is that this method needs consistent data on how those factors affect outcomes and assessment of which are the more successful across case studies.^[5]

Data collection

Process evaluations typically rely heavily on qualitative data, to cover both what the Magenta Book terms ‘objective’ and ‘subjective’ process issues.^[6]

Objective issues will cover the factual details of how the reforms have been implemented and operated. Typically, this makes use of administrative data. The availability and nature of any administrative data should always be considered at the design stage of an evaluation. This is because the data may not focus on the specific evaluation questions of interest or be collected regularly enough to provide useful evidence. It is important to ensure that administrative data includes all necessary records and transactions during the delivery of reform. Not only is this important for the process evaluation, but transactional data can be essential for any Cost-Benefit Analysis (CBA) or Value for Money (VfM) evaluation. The benefits of using administrative data are that it is both cost and time effective for the evaluator and participants, provides a more accurate record of processes than can sometimes be recalled in qualitative research and will also often cover the whole programme or population.

To explore the subjective issues on the perceptions of why and how well the reforms have been implemented and how they have operated, it is recommended that a future evaluator conducts (proportionate) qualitative data collection. As noted earlier, this could take the form of focus groups, interviews, and/or an open-ended survey. The exact method will depend on the needs of the evaluator, but all of these methods allow for views on the intentions, experiences, and self-reported critical assessment of how the reforms have been implemented and are operating in practice.

Inclusion of underrepresented groups

Similar to impact evaluation, any process evaluation can be designed to include particular consideration of underrepresented groups. In this context, the key evaluation questions would be to explore how well the research process considered, prioritised, and/or involved these groups (or a specific group(s)), and the experiences of different stakeholder groups. The primary benefit of this is to generate learnings for how this could be done better in future. Therefore, this would be most applicable to aspects of the reforms that will be repeated, such as the CIC process, or that are likely to have direct applicability to other major reform programmes in the near future.

Stakeholders that WPI Economics spoke to highlighted the importance of explicitly prioritising inclusivity and engaging with underrepresented groups at the outset, and throughout, the design and delivery of the reform process. This may also include ensuring decision making bodies or advisory groups are diverse and inclusive of relevant groups or for GME. Stakeholders felt that this would better ensure the reform outcomes served these groups well, and that there would be positive outcomes for these groups.

Therefore, the Scottish Government may choose to prioritise evaluation of the inclusivity of the reform process itself and to commission a specific evaluation to determine how successfully the reform process was informed by and considered the needs of underrepresented groups. This could be a standalone evaluation, or strand of an overarching process evaluation. This is also an area where a mid-stage evaluation may be particularly valuable to enable lessons to be fed into the continued design and delivery of the reform activities and improve the inclusion of underrepresented groups in the process as it moves forward. This would be aided by an explicit commitment to including underrepresented groups and representative bodies in the reform process, with detailed plans and targets for how the programme expects to do this set out at the outset of the reform process. The process evaluation would then be able to assess the extent to which this has been achieved and any key barriers and enablers.

Summary of process evaluation approaches

Process evaluation provides the opportunity to review and draw lessons from how the reform process took place. There are particular aspects of the reforms that we recommend are best suited to meaningful evaluation using this approach. For example, process evaluation can be an important tool to understand the stakeholder experience of different groups, which is relevant to the outcomes regarding stakeholders’ clarity and understanding of the reform process in particular. A process evaluation could also be created to understand how well the reform process incorporated and addressed the concerns of specific underrepresented groups, should this be a key evaluation priority.

Furthermore, this type of evaluation could also help understand how well the Education Reform Programme activities aided the creation of greater collaboration across the education system, and the process for delivering the changes to the curriculum, assessment, and qualifications system. These aspects of the reforms are highly likely to have lessons that can be actively used to improve future development of the education system as ongoing or repeated programmes of work.

The timeliness of process evaluations is particularly pertinent. Due to the extended timeline of the Education Reform Programme, it could be useful to employ a multi-staged evaluation. This could, for example, identify lessons from the process undertaken to develop and implement the curriculum changes, which can be applied to the subsequent phases of work for changes to the assessment and qualifications systems.

The table below summarises the recommended approach to pursue for undertaking the process evaluation of the Education Reform Programme and CQAR. This recommends the use of a mixed-methods approach for mid-stage as well as end-stage evaluations and to evaluate individual groups of activities or outputs, as needed. QCA is recommended as an end-stage evaluation method, allowing for analysis of the process of the reforms of the entire Programme.

Approach: Mixed-methods approach

Description: A mixed methods study combines quantitative and qualitative data collection and analysis in one study. Individually, these approaches can answer different questions, so combining them can provide more in-depth findings.^[7]

Challenges and limitations: The individual methods within this approach have certain limitations such as interviews and focus groups being resource intensive; requiring time to conduct and analyse; and not providing numerical estimates. There may also be a risk of bias in the views collected. Surveys are less useful for providing in-depth insight into an intervention and can have response-rate issues. Other methods may seem too onerous to participants and may be too resource intensive as well. ^[8]

Assessment: While the limitations hold true for individual methods, employing a mixed-methods approach is useful as no individual method can provide answers to all evaluation questions, consistent with the Magenta Book. A mixed methods process evaluation is recommended where there is interest in applying specific learnings to later parts of the reforms or similar future programmes. The evaluation can be designed in a manner that captures a holistic view of the activities of the Education Reform Programme, but if time and resources are limited, questions and methods will have to be prioritised and trade-offs between methods will be needed.

Approach: QCA

Description: This is a specific technical approach which can be used to help answer research questions within both process and impact evaluations. QCA provides a systematic approach to examining configurations of cases (groups of activities within the programme, in this case). It is commonly utilised in comparative studies and case-study research methodologies. Through QCA, data is interpreted qualitatively while assessing causal relationships among variables.^[9]

Challenges and limitations: A key aspect to the success of this method is availability of consistent data on how the causal factors affect the outputs and outcomes, which may not be available in a consistent format for the range of projects within the Education Reform Programme. This would also be resource intensive to collect through primary research.

Assessment: This method is used to compare multiple cases (activities and outputs, in this case) and systematically understand patterns of characteristics associated with desired or undesired outcomes based on qualitative knowledge. With sufficient resources, it could be a useful method to supplement other process and impact evaluation methods, given the complexity of the reform activities, programme structure and external factors, to deepen understanding of the factors of success that aided positive outcomes across different parts of the project.