Attainment Scotland Fund 2022-2026 evaluation: analytical plan - year 2 2023 -2024

Analytical Plan Year 2 (2023/24)

Purpose of the document

This document outlines the second annual evaluation plan for the Attainment Scotland Fund (ASF) 2022-26 and covers 2023-24, the second year of the refreshed Scottish Attainment Challenge. This is a companion document to the ASF Evaluation Strategy 2022-26, published in November 2022. The first Analytical Plan was published in February 2023. ^[1]

The document is intended to be accessible to all those with an interest in education in Scotland and beyond; including teachers, support workers, parents/carers, public bodies, and third sector organisations.

The ASF Evaluation

The ASF evaluation aims to provide learning about the overall implementation of the refreshed ASF and the extent to which progress has been made towards meeting intended outcomes articulated in the Scottish Attainment Challenge Logic Model in support of the refreshed programme mission.

The Evaluation Strategy for the Attainment Scotland Fund 2022-2026 ^[2] sets out the broad terms of the Scottish Government's approach to evaluating the ASF during this parliamentary term. The Strategy outlines four key strands of evaluation focus across the multi-year evaluation:

• Process evaluation to consider implementation of the refreshed Attainment Scotland Fund, including Strategic Equity Fund, Pupil Equity Fund and Care Experienced Children and Young People Fund;
• A thematic strand of evaluation which will respond to emerging system priorities and consider 'what works, for whom and in what circumstances'.
• Reporting on the National Improvement Framework (NIF) measures;
• Evaluation of the impact of ASF, through a combination of qualitative and quantitative measures.

This Analytical Plan

This second annual plan, continues a refreshed approach to evaluating progress in closing the poverty related attainment gap through the Attainment Scotland Fund. Priorities in the Analytical Plan include:

• Continuing to evaluate the implementation of the refreshed Scottish Attainment Challenge through research with a wider set of stakeholders.
• Quantitative reporting on core NIF Measures.
• Implement an impact evaluation plan following publication of the Impact Evaluation Feasibility Study undertaken in Year 1.
• Publish three thematic evaluation reports on: readiness to learn; families and communities; children and young person voice. Plans for year two will include a focus on learning and teaching outcomes from the logic model.

Progress of the ASF Evaluation in Year 1 (2022/23)

We have worked with the Evaluation Advisory Panel^[3] throughout the first year of the new Evaluation Strategy taking a flexible, adaptive, and responsive approach to building an evidence base. In the first year we have seen some adaptations and changes to the Plan which are noted in Table 1.

The key research activities/outputs from Year 1 were:

• A report on the implementation of the refreshed Scottish Attainment Challenge based on a survey of local authority attainment challenge leads.
• Publication of a report that summarises evidence around improvement in attainment and health and wellbeing, and the gap between pupils from the most and least deprived areas.
• Collaboration with National Programmes on engagement with Children and Young People on their views on readiness to learn.
• Collaboration with Education Scotland on the theme of Families and Communities.
• An impact evaluation feasibility study in collaboration with the Evaluation Advisory Panel.

Evaluation priorities for 2023/24 school year

Our priorities for 2023/24 are to continue to undertake a programme of analytical activity in support of the new Evaluation Strategy, with a focus on taking forward recommendations from the Impact Feasbility Study, reporting on the core NIF measures, continuing aspects of the process evaluation and ongoing exploration of existing and newly identified thematic areas.

Process: The evaluation will continue to focus on the implementation of the refreshed Scottish Attainment Challenge and new funding arrangements, developing insights on issues such as funding, guidance, Stretch Aims, planning and outcomes. In year 2, the focus of research activities will be schools and a wider set of stakeholders, which widens the focus from the local authority level in Year 1.

Impact: An Impact Evaluation Feasibility Study was undertaken in collaboration with the Evaluation Advisory Panel in Year 1. The study provides an in-depth consideration of the most promising potential approaches to assessing impact across Years 2 - 5. A summary of the process and options considered are contained at Appendix C. In year two, we will progress the development and initial implementation of a plan for impact assessment, in line with impact feasibility study recommendations.

NIF Attainment and Health and Wellbeing Measures: In order to maintain trend data and comparability, the core NIF measures continue to form the basis of quantitative reporting in Spring 2024 on the attainment gap. We will explore the feasbility of an expanded analysis of the NIF Attainment and Health and Wellbeing Measures.

Thematic: Analytical activity in Year 1 sought to explore 'what works, for whom and in what circumstances' on a number of priority themes including readiness to learn, families and communities, and children and young peoples engagement in decision-making/voice. We will publish a summary report and three separate companion reports on the thematic areas in February 2024. Whilst we continue to explore the thematic focus going forward with the Evaluation Advisory Panel, plans for year two will include developing a focus on learning and teaching outcomes from the logic model, and a continued focus on readiness to learn and children and children and young peoples voice in decision making. It is proposed that families and communities thematic focus will be complete in year two with the publication of the thematic report.

Our specific areas of activity are:

Section 2: Overview of the Analytical Plan

In order to address each of the four key evaluation strands over the duration of the Evaluation Strategy 2022-26, we have developed a high level overview of the analytical plan. This is outlined in Table 3 below. The detail of analytical activity during each year will be determined in line with the adaptive and flexible approach proposed in the Evaluation Strategy. The proposed activity for year 2, 2023/24 is set out in Appendix A.