Publication - Research and analysis

Attainment Scotland Fund Evaluation – Headteacher Survey 2020: technical report

Published: 17 Jun 2021

The Attainment Scotland Fund Evaluation: Headteacher Survey 2020 Technical Report provides details of the survey methodology and response, analysis and reporting, including a list of the survey questions, and the tabular results for the Headteacher Survey 2020.

Attainment Scotland Fund Evaluation – Headteacher Survey 2020: technical report
1. Survey methodology and response

1. Survey methodology and response

1.1. The Scottish Government commissioned the survey to include headteachers of schools in receipt of Challenge Authority, Schools Programme and/or Pupil Equity Funding. The overall aim was to gather information in order to build on learning from previous surveys to further improve operation of the ASF, and to maximise the impact of programmes supported by the Fund. This included the following specific objectives:

  • Provide insight on the experience of headteachers benefiting through each of the ASF streams during the 2019/20 school year, and understanding the impact of COVID-19 on ASF-supported approaches during March to June 2020;
  • Consider whether (and how) experiences have varied across key groups;
  • Build on longitudinal data to monitor changes over time; and
  • Provide evidence of what is working and what is not working well to inform ongoing delivery of the ASF.

1.2. This was the fifth Headteacher Survey since 2016. The survey scope and content have evolved over this period, and for the present survey this involved streamlining of existing questions to minimise the burden on schools, and a small number of new questions exploring the impact of COVID-19 on work to close the attainment gap. The main themes explored through the survey were:

  • Development of ASF-supported approaches including understanding of the challenges and barriers faced by pupils affected by poverty, capacity to select from the range of approaches that could be used to close the poverty-related attainment gap, and the extent to which achieving equity in education is embedded within school communities;
  • Use of data and evaluation, including views on skills and capacity within the school to use evidence to develop approaches and measure their impact;
  • Impact achieved in terms of closing the poverty-related attainment gap including how COVID-19 had affected progress to date, and views on the sustainability of progress and the focus on equity in education;
  • The extent to which ASF support has contributed to an increase in collaborative working, within and between schools, and with other agencies;
  • Views on processes around Pupil Equity Funding, including developing a PEF plan.

1.3. The survey sampling approach was modified from the 2019 survey to minimise the burden on the sector in the context of ongoing school building closures, while ensuring sufficient volume of responses to meet analysis requirements. On this basis, the survey was issued to headteachers of all Challenge Authority and Schools Programme schools, and 50% of those in receipt of Pupil Equity Funding only. The profile of schools across the three ASF streams and urban/rural geography is summarised below.

Survey population by ASF stream

  Challenge Authority Schools Programme PEF-only All
Urban area 525 49 392 966 (60%)
Small town 47 8 113 168 (10%)
Rural area 61 8 341 410 (25%)
Unknown 25 8 38 71 (4%)
Total 658 (41%) 73 (5%) 884 (55%) 1,615

Note: 'Unknown' urban/rural location is recorded where school postcode information could not be matched to geocoding data.

1.4. Urban/rural stratification of the survey sample was based on the 6-fold Scottish Government urban/rural classification[1]:

  • Urban area: schools in areas classified as '1: large urban' or '2: other urban';
  • Small town: schools in areas classified as '3: accessible small town' or '4. remote small towns'; and
  • Rural area: schools in areas classified as '5. accessible rural areas' or '6. remote rural areas'.

1.5. Consistent with previous surveys, survey invites were issued directly to schools, supported by promotion via Education Scotland and local authorities. The survey was issued in January 2021 and the fieldwork period extended to enable the broadest possible response, running to mid February 2021.

Survey response

1.6. A total of 420 responses were received by survey close, equivalent to an overall response rate of 27%. This represents a 20-point decrease since 2019, primarily due to a 30-point decrease in response from PEF-only schools (13% compared to 43% in 2019). As noted above, survey fieldwork took place in early 2021 during a period of school building closures and an associated increase in pressure on school resources. In this context, we very much appreciate those schools able to participate. Moreover, the volume of response is sufficient to produce robust results to inform the wider ASF evaluation (see 'Analysis and Reporting' later in this section for further detail).

Overview of survey response

ASF Stream Surveys issued Returns Response rate
Challenge Authority 658 278 42%
Schools Programme 73 24 33%
PEF-only 828 111 13%
Total 2,362 420 47%
Survey Response by ASF stream
Bar chart showing breakdown of survey responses by ASF funding stream and overall 2016 - 2020

1.7. The table below summarises the profile of survey respondents, and compares this with all schools in receipt of ASF support. In terms of the profile of respondents, the largest groups are Challenge Authority schools, primary schools, schools in urban areas, and schools with a middle or upper PEF allocation. This differs from the profile of all ASF-supported schools on a number of indicators, in part due to the change in sampling approach for the present survey (i.e. the reduced sample size for PEF-only schools):

  • Challenge Authority schools are over-represented and PEF-only schools under-represented. This is due to the 50% sample taken of PEF-only schools (compared to the 100% sample of CA schools), compounded by a lower response rate from PEF-only schools. Survey weighting has been used to correct for this response bias.
  • Schools with lower PEF allocations are under-represented, and those with higher allocations are over-represented. This is also in part due to the under-representation of PEF-only schools (which typically have lower PEF allocations) and over-representation of CA schools (which typically have higher allocations).
  • Schools in rural areas are under-represented, and schools in urban areas over-represented. Again, survey weighting has been used to correct for this.

Profile of survey respondents

Attainment Scotland Fund Respondents (n=420) All schools in receipt of ASF Differential
Challenge Authorities 58% 27% 31%
Schools' Programme 11% 3% 8%
PEF-only 31% 67% -36%
PEF allocation[2] Respondents (n=420) All schools in receipt of ASF Differential
Lower 10% 22% -12%
Middle 43% 42% 0%
Higher 40% 21% 19%
Unknown 7% 14% -
School sector Respondents (n=420) All schools in receipt of ASF Differential
Primary schools 80% 80% -1%
Secondary schools 15% 15% 1%
Special schools 5% 5% 0%
Unknown 0% 0% -
Urban/rural location Respondents (n=420) All schools in receipt of ASF Differential
Urban 64% 49% 15%
Small town 10% 10% -1%
Rural 15% 33% -18%
Unknown 11% 7% -

Analysis and reporting

1.8. Survey responses have been weighted by ASF stream and urban/rural location to adjust for response bias; all results presented in the remainder of this report are weighted. Base numbers for each survey question vary due to question non-response – i.e. they results exclude non-respondents to the question unless stated otherwise.

1.9. Survey analysis has used hypothesis tests with a 5% significance level to identify significant differences from previous survey findings, and across key respondent groups. These included:

  • ASF stream;
  • PEF allocation;
  • School sector; and
  • Urban/rural location.

1.10. Where variation across these groups is noted in the body of the survey report, this is based on a statistically significant difference between groups.

1.11. Survey data showed some inconsistency between responses and data on ASF support provided to schools - for example, 28 Challenge Authority respondents indicated that their school received only Pupil Equity Funding. The categorisation of respondents used in our analysis has been based on Scottish Government records rather than self-reporting.

1.12. The survey included a number of questions giving headteachers the opportunity to respond in their own words. This feedback has been coded into broad themes, and the results presented in the survey report. This includes the percentage of respondents coded into each theme – note that these percentages are based on those answering the question, and respondents can be assigned to multiple themes. Presentation of written feedback also includes direct quotes - this material has been edited for brevity and to ensure anonymity.