Programme for International Student Assessment (PISA) 2018: highlights from Scotland's results

Report covering Scotland's performance in the Programme for International Student Assessment (PISA) 2018, covering reading, maths and science.


2. How Reading is Assessed

1. PISA assesses reading literacy, as opposed to reading. Reading is often interpreted as reading aloud or simply converting text into sounds; reading literacy, on the other hand, is a broader set of competencies that allows readers to engage with written information, presented in one or more texts, for a specific purpose.

2. To do so, readers must understand what is written and integrate this with their pre-existing knowledge. They must examine the author's (or authors') point of view and decide whether the text is reliable and truthful, and whether it is relevant to their goals or purpose.

3. PISA also recognises that reading is a daily activity for most people, and that education systems need to prepare students to be able to adapt to the variety of scenarios in which they will need to read as adults. These scenarios range from their own personal goals and development initiatives, to their interactions at work, with public entities, in online communities and with society at large. It is not enough to be a proficient reader; students should also be motivated to read and be able to read for a variety of purposes.

4. All of these considerations are reflected in the PISA 2018 definition of reading literacy:

Reading literacy is understanding, using, evaluating, reflecting on and engaging with texts in order to achieve one's goals, to develop one's knowledge and potential, and to participate in society.

5. Below, we summarise key features of the OECD's framework for assessing reading.

The PISA 2018 framework for assessing reading

For PISA 2018, the reporting subscales are:

1) Locating information, which is composed of tasks that require students to search for and select relevant texts, and access relevant information within texts;

2) Understanding, which is composed of tasks that require students to represent the explicit meaning of texts as well as integrate information and generate inferences; and

3) Evaluating and reflecting, which is composed of tasks that require the student to assess the quality and credibility of information, reflect on the content and form of a text and detect and handle conflict within and across texts.

6. Questions are constructed to test each of these categories, and at varying levels of difficulty, in order to identify a student's ability. Their score corresponds to levels of ability, which are summarised in Table 2.1 below. Example questions and how they were adapted for computer-based assessment are provided in Annex C of Volume I of the OECD report.

Table 2.1: Proficiency levels in reading, and what they mean

Level

Lower score limit

Characteristics of tasks

6

698

Readers at Level 6 can comprehend lengthy and abstract texts in which the information of interest is deeply embedded and only indirectly related to the task. They can compare, contrast and integrate information representing multiple and potentially conflicting perspectives, using multiple criteria and generating inferences across distant pieces of information to determine how the information may be used. Readers at Level 6 can reflect deeply on the text's source in relation to its content, using criteria external to the text. They can compare and contrast information across texts, identifying and resolving inter-textual discrepancies and conflicts through inferences about the sources of information, their explicit or vested interests, and other cues as to the validity of the information. Tasks at Level 6 typically require the reader to set up elaborate plans, combining multiple criteria and generating inferences to relate the task and the text(s). Materials at this level include one or several complex and abstract text(s), involving multiple and possibly discrepant perspectives. Target information may take the form of details that are deeply embedded within or across texts and potentially obscured by competing information.

5

626

Readers at Level 5 can comprehend lengthy texts, inferring which information in the text is relevant even though the information of interest may be easily overlooked. They can perform causal or other forms of reasoning based on a deep understanding of extended pieces of text. They can also answer indirect questions by inferring the relationship between the question and one or several pieces of information distributed within or across multiple texts and sources. Reflective tasks require the production or critical evaluation of hypotheses, drawing on specific information. Readers can establish distinctions between content and purpose, and between fact and opinion as applied to complex or abstract statements. They can assess neutrality and bias based on explicit or implicit cues pertaining to both the content and/or source of the information. They can also draw conclusions regarding the reliability of the claims or conclusions offered in a piece of text. For all aspects of reading, tasks at Level 5 typically involve dealing with concepts that are abstract or counterintuitive, and going through several steps until the goal is reached. In addition, tasks at this level may require the reader to handle several long texts, switching back and forth across texts in order to compare and contrast information

4

553

At Level 4, readers can comprehend extended passages in single or multiple-text settings. They interpret the meaning of nuances of language in a section of text by taking into account the text as a whole. In other interpretative tasks, students demonstrate understanding and application of ad hoc categories. They can compare perspectives and draw inferences based on multiple sources. Readers can search, locate and integrate several pieces of embedded information in the presence of plausible distractors. They are able to generate inferences based on the task statement in order to assess the relevance of target information. They can handle tasks that require them to memorise prior task context. In addition, students at this level can evaluate the relationship between specific statements and a person's overall stance or conclusion about a topic. They can reflect on the strategies that authors use to convey their points, based on salient features of texts such as titles and illustrations. They can compare and contrast claims explicitly made in several texts and assess the reliability of a source based on salient criteria. Texts at Level 4 are often long or complex, and their content or form may not be standard. Many of the tasks are situated in multiple-text settings. The texts and the tasks contain indirect or implicit cues.

3

480

Readers at Level 3 can represent the literal meaning of single or multiple texts in the absence of explicit content or organisational clues. Readers can integrate content and generate both basic and more advanced inferences. They can also integrate several parts of a piece of text in order to identify the main idea, understand a relationship or construe the meaning of a word or phrase when the required information is featured on a single page. They can search for information based on indirect prompts, and locate target information that is not in a prominent position and/or is in the presence of distractors. In some cases, readers at this level recognise the relationship between several pieces of information based on multiple criteria. Level 3 readers can reflect on a piece of text or a small set of texts, and compare and contrast several authors' viewpoints based on explicit information. Reflective tasks at this level may require the reader to perform comparisons, generate explanations or evaluate a feature of the text. Some reflective tasks require readers to demonstrate a detailed understanding of a piece of text dealing with a familiar topic, whereas others require a basic understanding of less-familiar content. Tasks at Level 3 require the reader to take many features into account when comparing, contrasting or categorising information. The required information is often not prominent or there might be a fair amount of competing information. Texts typical of this level may include other obstacles, such as ideas that are contrary to expectation or negatively worded.

2

407

Readers at Level 2 can identify the main idea in a piece of text of moderate length. They can understand relationships or construe meaning within a limited part of the text when the information is not prominent by producing basic inferences, and/or when the information is in the presence of some distracting information. They can select and access a page in a set based on explicit though sometimes complex prompts, and locate one or more pieces of information based on multiple, partly implicit criteria. Readers at Level 2 can, when explicitly cued, reflect on the overall purpose, or on the purpose of specific details, in texts of moderate length. They can reflect on simple visual or typographical features. They can compare claims and evaluate the reasons supporting them based on short, explicit statements. Tasks at Level 2 may involve comparisons or contrasts based on a single feature in the text. Typical reflective tasks at this level require readers to make a comparison or several connections between the text and outside knowledge by drawing on personal experience and attitudes.

1a

335

Readers at Level 1a can understand the literal meaning of sentences or short passages. Readers at this level can also recognise the main theme or the author's purpose in a piece of text about a familiar topic, and make a simple connection between several adjacent pieces of information, or between the given information and their own prior knowledge. They can select a relevant page from a small set based on simple prompts, and locate one or more independent pieces of information within short texts. Level 1a readers can reflect on the overall purpose, gist and adjunct information in simple texts containing explicit cues. Most tasks at this level point to relevant factors in the task and in the text.

1b

262

Readers at Level 1b can evaluate the literal meaning of simple sentences. They can also interpret the literal meaning of texts by making simple connections between adjacent pieces of information in the question and/or the text. Readers at this level can scan for and locate a single piece of prominently placed, explicitly stated information in a single sentence, a short text or a simple list. They can access a relevant page from a small set based on simple prompts when explicit cues are present. Tasks at Level 1b explicitly direct readers to consider relevant factors in the task and in the text. Texts at this level are short and typically provide support to the reader, such as through repetition of information, pictures or familiar symbols. There is minimal competing information.

1c

189

Readers at Level 1c can understand and affirm the meaning of short, syntactically simple sentences on a literal level, and read for a clear and simple purpose within a limited amount of time. Tasks at this level involve simple vocabulary and syntactic structures.

Source: OECD

Contact

Email: keith.dryburgh@gov.scot

Back to top