Programme for International Student Assessment 2025: national report for Scotland

3 Strengths and limitations of PISA

PISA is one of the largest and most rigorously designed international studies of educational outcomes. It provides valuable evidence on the knowledge, skills and experiences of 15 year olds across a wide range of countries and economies, enabling comparisons both internationally and over time.

PISA’s principal strengths are the consistency of its international methods, the breadth of participating education systems, its ability to examine changes over more than two decades, and the combination of performance data with information about pupils’ backgrounds and experiences. This allows Scotland’s results to be considered both in an international context and alongside wider characteristics of the education system. However, as with any large-scale assessment, its findings should be interpreted in light of a number of important limitations.

3.1 Coverage of the target population

PISA is designed to assess a representative sample of 15 year old pupils who are enrolled in eligible educational institutions. It does not assess all young people of that age. Young people who are not enrolled in an eligible educational institution, such as those who are home educated, are outside the PISA target population. Pupils who meet internationally agreed exclusion criteria are also not included in the assessed population.

OECD standards allow a small proportion of pupils to be excluded where participation would not be feasible or appropriate. This includes, for example, pupils with severe intellectual disabilities, significant functional needs that would prevent meaningful participation, or insufficient proficiency in the language of assessment.

Scotland’s exclusion rate in PISA has historically been higher than the OECD’s recommended maximum of 5 per cent. In PISA 2025, Scotland’s overall exclusion rate was 7.89 per cent. Most excluded pupils were recorded as having additional support needs which, in the judgement of schools and in line with OECD guidance, meant that participation would not have been appropriate or feasible.

Comparisons of exclusion rates between countries should be treated with caution. Education systems differ in how additional support needs are identified, recorded and supported, and there is no single internationally standardised system for classifying these needs. Scotland uses a broad definition of additional support needs, covering needs arising from a wide range of circumstances and not only disabilities or learning difficulties. Differences in how needs are identified and recorded across education systems mean that exclusion rates should not be interpreted as directly comparable measures of the prevalence of additional support needs.

Scotland’s exclusion rate for PISA 2025 was reviewed through the OECD’s international quality assurance and adjudication processes. The OECD’s Adjudication Group noted that Scotland’s rate was above the recommended threshold, but concluded that Scotland’s results were suitable for inclusion in international reporting and for comparisons over time. Scotland was not unusual in exceeding the exclusion-rate standard: around one quarter of adjudicated PISA 2025 participants also exceeded it. The Adjudication Group nevertheless noted the need to reduce within-school exclusions in future cycles.

Readers should therefore bear in mind that the results relate to the eligible participating population and may not fully reflect the performance of all 15 year olds in Scotland. Differences in exclusion rates between countries may also affect international comparability, particularly when interpreting small differences in performance between education systems.

Because excluded pupils are not assessed, it is not possible to know precisely how their inclusion would have affected Scotland’s results. However, if excluded pupils would, on average, have achieved lower scores than participating pupils, Scotland’s overall performance estimates may be somewhat higher than would have been observed had all pupils been included.

3.2 School and pupil participation

Like all sample surveys, PISA relies on the participation of sampled schools and pupils. Some sampled schools decline to participate, and some selected pupils do not take part on the day of assessment.

OECD technical standards specify minimum participation rates and require countries to undertake procedures to maximise participation. Statistical weighting adjustments are also applied to reduce the risk of bias arising from non-response.

However, it is not possible to eliminate entirely the possibility that participating schools and pupils differ systematically from those who do not participate. For example, if lower-performing schools or pupils are less likely to participate, results may not fully represent the wider population.

For this reason, a degree of uncertainty remains regarding how closely the achieved sample reflects the full population of eligible pupils.

3.3 International comparability

A major strength of PISA is its ability to support international comparison. However, comparisons between countries should not be interpreted as comparisons between identical educational systems. Education systems differ substantially in their curricula, school structures, educational policies, social contexts, demographics and cultural expectations.

Extensive translation, verification and quality assurance procedures are used to maximise comparability across countries. Nevertheless, research has shown that some assessment items and questionnaire measures may operate differently across cultural and linguistic contexts.

Similarly, many questionnaire measures rely on self-reported information from pupils, teachers and school leaders. Differences in interpretation, response styles and cultural norms may influence responses, particularly when comparing attitudes, perceptions and behaviours across countries.

Questionnaire findings are also based on pupils’ or school leaders’ reports rather than independently observed behaviour or conditions. They should therefore be interpreted as measures of reported experiences, perceptions and circumstances.

3.4 Measurement uncertainty

PISA results are estimates rather than precise measures of pupil performance. Individual pupils answer only a subset of all available assessment questions and scores are estimated using advanced statistical models rather than calculated directly from the number of correct answers.

This approach allows a broad range of content to be assessed while reducing burden on individual pupils. Statistical models are then used to estimate performance across the full assessment, taking account of the fact that each pupil answers only a subset of the available questions.

PISA also assesses a limited range of knowledge and skills. While reading, mathematics and science literacy are important educational outcomes, they do not capture all aspects of learning, achievement, wellbeing, creativity, or personal development. PISA also does not assess how well pupils have learned a specific curriculum. Instead, it measures how effectively they can apply their knowledge and skills to unfamiliar problems and real-world situations. The findings therefore provide evidence about broader knowledge and skills rather than mastery of particular curriculum content.

PISA is also a low-stakes assessment for individual pupils: their results do not contribute to school qualifications or affect their educational progression. Pupils may therefore differ in the effort or motivation they bring to the assessment, and these differences may vary between individuals and education systems.

3.5 Trend comparisons over time

PISA is specifically designed to support comparisons over time, with common assessment items and statistical linking procedures used to maintain comparable scales between cycles. Nevertheless, changes in assessment frameworks, item content or delivery methods can introduce additional uncertainty into some comparisons.

Changes in participation rates, exclusion rates or the characteristics of participating pupils and countries may also influence comparisons between survey cycles.

Consequently, apparent changes in performance should not automatically be interpreted as evidence of genuine improvement or decline unless they are statistically significant once all sources of uncertainty, including linking error, have been taken into account.

3.6 Interpretation and use of rankings

International rankings are often used to summarise PISA results. However, rankings should be interpreted with caution. Many countries achieve scores that are not statistically significantly different from one another, meaning that small differences in rank may not represent meaningful differences in educational performance.

For this reason, the report focuses primarily on statistical comparisons rather than ranking countries from highest to lowest based on their average scores. More emphasis should be placed on the size and significance of differences in performance than on a country’s precise position in international league tables.

3.7 PISA as one source of evidence

PISA provides an important source of evidence on educational outcomes and experiences, but it should not be viewed in isolation. Educational performance is multifaceted and no single assessment can provide a complete picture of the strengths and challenges within an education system.

PISA is a cross-sectional survey, meaning that it assesses different groups of 15 year olds in each cycle rather than following the same pupils over time. As a result, PISA can identify patterns and trends but cannot by itself determine the causes of changes in performance.

PISA can identify associations between pupil characteristics, experiences and outcomes, but these should not be interpreted as evidence that one factor causes another. For example, an association between a particular school experience and higher or lower performance may reflect other differences between the pupils or schools concerned. Findings from the questionnaire data should therefore be interpreted as evidence of relationships and patterns rather than causal effects.

The findings presented in this report should therefore be considered alongside evidence from national qualifications, curriculum assessments, inspection evidence, administrative data, research studies and other international surveys. Taken together, these sources provide a more complete understanding of educational performance and outcomes in Scotland.