Attainment Scotland Fund evaluation: third interim report - year 4

Chapter 6 Discussion and conclusions

6.1 As described in the introductory section of the report, the evaluation aims to provide learning about the overall implementation of the ASF and the extent to which the aims of ASF are being met in terms of closing the poverty-related attainment gap.

6.2 The ASF evaluation report 2019 has sought to bring together a range of evidence which provides the basis for assessing the extent to which progress has been made towards the overall aims of the fund. The concluding section:

• draws together the key findings related to the assessment of progress and learning about the process of implementing the Fund, together with consideration of contextual/background material (where appropriate to support this assessment)
• provides final reflections on progress to date
• offers some commentary on the evaluation and its future direction

Key findings

6.3 There has been a move away from a focus on interventions to approaches to close the poverty-related attainment gap. There is increasing evidence of a focus on broader approaches implemented around literacy, numeracy, and health and wellbeing to close the poverty-related attainment gap rather than at the intervention level. This includes local-authority wide developments such as whole school nurture approaches.

6.4 Approaches to closing the poverty-related attainment gap are being refined based on improvement and use of evidence. Enhancements or adaptations in approaches to literacy, numeracy, health and wellbeing introduced to close the poverty-related attainment gap were based on the increasing use of a broad range of data, a focus on measuring impact, and a focus on building sustainability.

6.5 Tailoring to local context and sufficient resources are key factors associated with success. The ability to implement approaches relevant to the local/school context and having sufficient teaching and staffing resources were the most commonly cited factors contributing to progress. Staff time/workload and reduction in other resources were seen as the main factors limiting progress.

6.6 Culture change and ethos. There is evidence of systemic change in terms of culture and ethos towards development of a shared understanding of barriers faced by pupils and families, approaches to equity becoming more embedded within the school, and a more collaborative system.

6.7 Perceptions of success. There are continuing positive messages in terms of perceptions of success:

• Nine out of ten (91%) of headteachers are reported to have seen an improvement in closing the poverty-related attainment gap as a result of ASF supported approaches, an increase of 13 points since 2017
• Nearly all headteachers (98%) expect to see improvement in closing the gap over the next five years

6.8 Clear links between perceptions of success and positive culture change/ethos. Headteachers most likely to report improvement in closing the poverty-related attainment gap were those who had seen a change in culture or ethos (such as embedding approaches to equity, broader strategies, more collaborative working) and where there was improved understanding of barriers faced by pupils and families.

6.9 Clear links between perceptions of success and use of evidence in developing and evaluating ASF supported approaches. Headteachers most likely to have seen an improvement as a result of ASF supported approaches were those who feel confident using evidence to inform the development of their approach, and always use evidence to measure the impact of these approaches.

6.10 Variability across funding streams and urban/rural contexts. This included variations of perceptions from headteachers across the three funding streams in respect of reported progress in closing the poverty-related attainment gap, with Schools Programme respondents most likely to report an improvement, whereas PEF-only schools and particularly those with a lower PEF allocation were least likely to report an improvement.

6.11 Funding continues to provide a focus. Whilst there are positive perceptions of the sustainability of focus on closing the attainment gap to some extent, it is recognised that funding is key in ensuring the resources to support improvements.

6.12 Closing the attainment gap – the quantitative measures.

The analysis presented in Section 5 indicates that on a number of measures, there is progress in closing the attainment gap, although this is a varied picture depending on the measure under consideration. Progress in closing the attainment gap is assessed through a 'basket' of measures. This basket of measures, introduced at the start of the ASF evaluation, looks at various National Improvement Framework attainment measures and considers how the data collected and the trends in data over time can provide illumination on the gap overall – by subject, by age, and by local authority. Quantitative measures of the attainment gap do not yet show a consistent pattern of change. A summary is provided below.

Summary of Quantitative Measures

Achievement of Curriculum for Excellence Levels (ACEL)

6.13 In Primary schools, the proportion of primary pupils achieving the expected level in both literacy and numeracy steadily increased between 2016/17 and 2018/19 in Challenge Authorities, non-Challenge Authorities and in Scotland overall.

6.14 The gap between the proportion of Primary pupils from the most and least deprived areas that have achieved the expected level in literacy narrowed for Challenge Authorities, non-Challenge Authorities and Scotland overall between 2017/18 and 2018/19. For numeracy the gap widened slightly for Challenge Authorities and non-Challenge Authorities between 2017/18 and 2018/19, and remained the same at Scotland level.

6.15 In secondary schools, S3 pupils achieving the expected level in literacy has decreased slightly in Challenge Authorities between 2016/17 and 2018/19, but increased in non-Challenge Authorities and at Scotland level. In numeracy, the proportion achieving expected levels has risen in Challenge Authorities, non-Challenge Authorities and at Scotland level.

6.16 The gap between the proportion of Secondary pupils from the most and least deprived areas that have achieved the expected level in numeracy narrowed across Challenge Authorities, non-Challenge Authorities and at Scotland level. The gap in literacy narrowed in non-Challenge Authorities, but has widened slightly in Challenge Authorities and at Scotland level.

Senior Phase (SCQF) levels

6.17 The size of the gap between the proportion of school leavers from the most and least deprived areas attaining one or more pass varies by SCQF Level. At SCQF level 5, the size of the gap has narrowed slightly for Challenge Authorities and Scotland level and widened slightly for non-Challenge Authorities. At SCQF levels 6 and 7, the gap has narrowed across Challenge Authorities, at Scotland level and for non-Challenge Authorities.

Participation Measure

6.18 The Annual Participation Measure reports the percentage of young adults (16-19 year olds) participating in education, training or employment. Between 2017 and 2019, there was an overall reduction in the participation gap between those living in the most deprived areas compared to those living in the least deprived areas. This was due to an increase in the rate of participation amongst those within the most deprived areas compared to a slight decrease amongst those in the least deprived areas. Six of the nine Challenge Authorities recorded a slight increase between Year 3 and Year 4 of the ASF. This reflects a similar trend to 2018 when seven out of the nine recorded an increase between Year 2 and Year 3 of the ASF.

Attendance and Exclusion

6.19 The gap in attendance rates increased over time for primary school children and, whilst the effect is greater in secondary schools, the gap has remained the same in years 2016/17 – 2018/19. Whilst the attendance of secondary school pupils from the least deprived areas remained fairly stable over time, the attendance rate of the most deprived pupils decreased.

6.20 Overall, exclusion rates were higher for pupils from the most deprived areas compared to those from the least deprived. Secondary schools had a higher exclusion rate than primary schools and the gap in exclusion rates was also higher in secondary schools. Due to suppression of figures it is difficult to draw conclusions from the exclusion rate figures.

6.21 Therefore, the assessment of progress in terms of whether the gap has been closed is nuanced and impacted by many contextual factors as described in Section Five. Given the timescales for some data collections, it can be difficult in the short to medium term to assess the extent of progress. Some impacts may not emerge in terms of data until the longer term (for example, Level 5 and 6 qualifications data will not emerge for a considerable number of years for current primary pupils).

6.22 Reported impacts/perceptions of success currently present a more positive picture of progress than are emerging through quantitative measures.

Reflections on progress to date

6.23 The strategic aim of the SAC is 'closing the poverty-related attainment gap between children and young people from the least and most disadvantaged communities' and at a wider policy level links to the National Performance Framework shared aim of improving outcomes for children and young people. As outlined in the introductory section, the SAC logic model articulates the following long-term outcomes:

Long Term Outcome 1: Embedded and sustained practices related to addressing the impact of the poverty-related attainment gap

Long Term Outcome 2: All children and young people are achieving the expected or excellent educational outcomes, regardless of their background

Long Term Outcome 3: An education system which is aspirational, inclusive in practice and approaches for all including teachers, parents and carers, children and young people

Long Term Outcome 4: Closing the attainment gap between the most and least disadvantaged young people

6.24 The ASF seeks to support progress towards the strategic aim and associated long term outcomes of SAC, alongside the wider range of national programmes and initiatives which form the SAC. This report has outlined evidence of progress towards achieving these long term outcomes.

6.25 The evidence presented in the report has highlighted positive progress in a number of areas, not least with regard to clear signs of the development of embedded and sustained practices related to addressing the impact of the poverty-related attainment gap, and positive developments across the education system towards aspirational, inclusive practice and approaches.

6.26 In terms of evidence related to closing the poverty-related attainment gap between the most and least disadvantaged young people, there are aspects which indicate positive progress, particularly in terms of the reported evidence of impact as outlined in the key findings above.

6.27 However, the overall trends emerging from the attainment data demonstrate the challenge which remains related to closing the poverty-related attainment gap between the most and least deprived pupils. This reaffirms that the commitment to closing the poverty-related attainment gap is a long term one with a need to work in partnership to facilitate, broker and support action to maximise the progress towards closing the poverty-related attainment gap.

6.28 This evaluation report focused on the 2018/19 academic year and as such presents evidence introduced prior to the introduction of the Maximising Progress priorities at the start of the 2019/20 academic year. Whilst in this respect this report is retrospective, the evidence gathered points to the importance of continued focus on Maximising Progress.

Evaluation and further research: reflections and future direction

6.29 The section below will seek to address the following questions:

• What are the limitations of the evaluation approach and its scope?
• How do we assess the strength of evidence, and the balance of qualitative versus quantitative evidence gathered?
• What are the gaps in the research evidence and how should these gaps be addressed?
• What areas are emerging for further consideration?

6.30 As described in the introduction to this report, the evidence presented in this report is based on the Year 3 and 4 Evaluation Strategy. A review of the current Evaluation Strategy is being undertaken with the aim of developing a refreshed Evaluation Strategy for the next two years of ASF. In setting out the assessment in this report, it is noted that the scope of the evaluation is limited to addressing the core evaluation questions and so is not comprehensive in its approach. It must also be recognised that it remains difficult to assess the reasons behind any observed improvement in attainment or closing the poverty-related attainment gap, and whether these changes have occurred as a direct result of the fund. However, the evaluation helps to support our understanding of factors which are related to these improvements, and the contribution the fund made to the realisation of these factors.

6.31 The evaluation is necessarily high level and may not capture the full picture at the local level. It is however recognised that there is considerable evidence at the local level which is not reflected in the national evaluation to date. Opportunities to increasingly reflect this in the evaluation going forward are being considered as part of the evaluation strategy refresh which is currently taking place.

6.32 It is also recognised that there are a number of aspects of analysis which it is intended to strengthen in future evaluation. It is recognised that the balance of quantitative and qualitative research requires to be addressed in future years.

6.33 A key issue for consideration for the future direction of the evaluation is the impact of Covid-19. This will include consideration of how best to gather data on the 2019/20 academic year given the impact of Covid-19, as well as exploring the impact of Covid-19 on both process and impact of the ASF in seeking to close the poverty-related attainment gap.

6.34 Ensuring the evaluation reflects the views of wider stakeholders is also a further aspect for consideration. For example, going forward we will seek to ensure the voices of parents and pupils feed in to the evaluation process more comprehensively. In addition, involvement of stakeholders representing communities, third sector and other partners will be considered in the evaluation strategy refresh.

6.35 It will be important that the 'beyond the school' context in terms of closing the poverty-related attainment gap is more fully recognised and explored in future evaluation, in particular the role of families (including parental involvement and engagement) and communities.

6.36 Another aspect for strengthened focus which will be considered in the evaluation strategy refresh will relate to children's health and wellbeing.

6.37 The key importance of focusing on attendance in continuing to support progress on closing the poverty-related attainment gap suggests a need to explore issues affecting attendance in greater detail going forward.