Programme for International Student Assessment (PISA 2022): Scotland's results - highlights

Annex 1: Background and Methodology

What is PISA?

143. PISA is an assessment of 15 year-olds' skills carried out under the auspices of the OECD. The programme runs every three years across all OECD members and a variety of partner countries. Scotland has participated in all eight surveys since the first wave of testing in 2000.

144. Each survey cycle focusses on one of three domains: reading, mathematics and science. In 2022 the main domain was maths, with reading and science as subsidiary domains. Data and analysis on creative thinking (the "innovative domain" in PISA 2022) will be published during 2024.

Who participates?

145. Around 690,000 students participated in the study worldwide, representing about 32 million 15 year olds. In 2022, 81 countries and economies participated in PISA.

OECD member countries in PISA 2022

• Australia
• Austria
• Belgium
• Canada
• Chile
• Colombia
• Costa Rica
• Czech Republic
• Denmark
• Estonia
• Finland
• France
• Germany
• Greece
• Hungary
• Iceland
• Ireland
• Israel
• Italy
• Japan
• Korea
• Latvia
• Lithuania
• Mexico
• Netherlands
• New Zealand
• Norway
• Poland
• Portugal
• Slovak Republic
• Slovenia
• Spain
• Sweden
• Switzerland
• Türkiye
• United Kingdom (excluding Scotland)
• United States

Partner countries and economies in PISA 2022

• Albania
• Argentina
• Baku (Azerbaijan)
• Brazil
• Brunei Darussalam
• Bulgaria
• Cambodia
• China (People's Republic of)
• Croatia
• Dominican Republic
• El Salvador
• Georgia
• Guatemala
• Hong Kong (China)
• Indonesia
• Jamaica
• Jordan
• Kazakhstan
• Kosovo
• Lebanon
• Macao (China)
• Malaysia
• Malta
• Moldova (Republic of)
• Mongolia
• Montenegro
• Morocco
• North Macedonia
• Palestinian Authority
• Panama
• Paraguay
• Peru
• Philippines
• Qatar
• Romania
• Saudi Arabia
• Serbia
• Singapore
• Chinese Taipei
• Thailand
• Ukraine
• United Arab Emirates
• Uruguay
• Uzbekistan
• Viet Nam

Countries and economies in previous cycles

• Algeria
• Azerbaijan
• Beijing (China)
• Belarus
• Bosnia and Herzegovina
• Guangdong (China)
• Himachal Pradesh (India)
• Jangsu (China)
• Kyrgyzstan
• Lebanon
• Liechtenstein
• Luxembourg
• Mauritius
• Miranda (Venezuela)
• Russian Federation
• Shanghai (China)
• Tam il Nadu (India)
• Trinidad and Tobago
• Tunisia
• Zhejiang (China)

146. The United Kingdom is a member state of the OECD and its results are published in the main OECD publication. Scotland participates as an "adjudicated region", meaning that its results have full quality assurance from the survey contractors appointed by the OECD, and can publish its results separately. Within the UK, England, Wales and Northern Ireland have boosted samples as "non-adjudicated regions" which means they are able to produce country-level analysis within their reports. Regional results are published as annexes to the main OECD volumes.

147. Survey fieldwork is carried out separately in each participating state by "National Centres" according to strict quality standards set by the OECD.

What does PISA measure?

148. PISA seeks to measure skills which are necessary for participation in society. Accordingly, it assesses how students apply the skills they have gained to the types of problem they may encounter in work or elsewhere. Pupils are assessed at the age of 15 as this is regarded as a reasonable point at which to test the impact of compulsory education throughout the developed world. After this point students will typically move onto more specialised studies or enter the labour market. Box 1.1 contains the definitions of the domains tested by PISA.

Box 1.1: The PISA domains and their definition

Mathematical literacy is defined an individual's capacity to reason mathematically and to formulate, employ, and interpret mathematics to solve problems in a variety of real-world contexts. It includes concepts, procedures, facts and tools to describe, explain and predict phenomena. It assists individuals to know the role that mathematics plays in the world and to make the well-founded judgments and decisions needed by constructive, engaged and reflective 21st-century citizens.

Reading literacy is defined as understanding, using, evaluating, reflecting on and engaging with texts in order to achieve one's goals, to develop one's knowledge and potential, and to participate in society.

Science literacy is defined as the ability to engage with science-related issues, and with the ideas of science, as a reflective citizen. A scientifically literate person is willing to engage in reasoned discourse about science and technology, which requires the competencies to explain phenomena scientifically, evaluate and design scientific enquiry, and interpret data and evidence scientifically.

149. We have included some details on how mathematics, the main focus of the 2022 PISA survey, was assessed in Annex 2. Further details of how each domain was assessed can be found in the OECD volumes published on the PISA website, www.oecd.org/pisa.

150. The assessments are also supplemented by background questionnaires. Pupils are asked about their motivations for study, attitudes to school, beliefs about science, studying and their socio-economic background. Headteachers are asked about the challenges facing their schools, organisation and factors that they believe affect their students' performance.

The survey in Scotland

151. The PISA survey was managed by an international consortium led by ETS. The Consortium developed the tests, questionnaires and survey documentation and ensured that all participating countries met quality standards. In Scotland, the National Foundation for Educational Research (NFER) was the "National Centre", responsible for local adaptations to the surveys, and administering the test in schools.

152. The school sample was randomly selected by NFER following submission of sampling forms to the consortium. The sample was stratified on the basis of previous exam performance (split into five categories), whether schools were publicly funded or independent, urban/rural location and school size, and whether schools were single-sex or mixed.

153. The survey was carried out in Scotland between 18 October 2022 and 22 December 2022. In total, 117 secondary schools participated in the survey. 106 of these were from the main sample (87.5 per cent response rate), and 11 from the back-up samples (resulting in a 96.4 per cent weighted participation rate after replacements were added in). This exceeded the OECD's minimum standard of 85 per cent participation.

154. Within each school 40 students were randomly sampled by NFER using software supplied by the Consortium. In total 4,652 students were drawn in the sample. Schools were able to withdraw a certain number of students where it was deemed that participation would be difficult due to additional support needs or language issues. Similarly students that had left the school in the interim were not considered part of the target sample. In total 4,115 students were deemed eligible participants. Of these a total of 3,257 students took part, with the balance being those who did not wish to take part (both students and their parents were given the opportunity to opt out of the survey), and those who were absent on the day of the test.

155. Students who participated in the assessments were mainly split between S4 (53.5% of the total sample) and S5 (46.3%). This compares to a 50/50 split between S4 and S5 in PISA 2018, which means that students participating in PISA 2022 are more likely to have been in S4 than in PISA 2018.

156. Test administrations were carried out as specified by the PISA consortium. However, lower than expected levels of student participation proved to be an issue during the main PISA 2022 testing window. This was addressed, with the agreement of the PISA consortium, by extending the period for follow-up sessions to the end of the Christmas term. Nevertheless, Scotland narrowly missed the student participation technical standard of 80 per cent participation with a weighted student participation rate of 79.4 per cent. Lower student participation proved to be an issue internationally in the PISA 2022 post-Covid-19 survey.

157. In order to establish whether the lower student participation rates in Scotland had an impact on the data, the PISA consortium requested that Scotland, along with other countries^[2], perform a student level non-response bias analysis (NRBA). The details and outcome of the NRBA are provided in the Main Survey Conduct Report which can be found here: PISA Research - NFER. The outcome of the NRBA is that the results for Scotland (and for the UK where they are combined) will be annotated in the OECD international report to indicate that Scotland submitted a technically strong analyses, which indicated that more than minimal bias was most likely introduced in the estimates due to low response rates. This bias was in an upward direction and of about 0.1 standard deviations (estimated to approximately equate to 9 or 10 points). The OECD also indicated that the PISA 2022 data for Scotland were comparable to previous cycles.

158. The assessment was administered in Scotland by computer. This was achieved using the existing facilities in schools with the support of school and Local Authority ICT services.

159. The software delivery system was provided by the international consortium. The assessment was administered in two one-hour sessions, with a further 35 minutes for the background questionnaire. Students spent one hour on the mathematics assessment (composed of a core stage followed by two stages of either greater or lesser difficulty) plus one hour on one or two other subjects – mathematics and science.

160. As in all previous cycles, there was a survey of headteachers within schools, which asked about their views on school organisation, teaching staff and resources. 102 headteachers responded – a response rate of 85 per cent.