Validity and reliability of measurements of the National Standardised Assessments: FOI release

Information request and response under the Freedom of Information (Scotland) Act 2002.


FOI reference: FOI/18/02228
Date received: 21 August 2018
Date responded: 18 September 2018
 
Information requested
 
  1. What is the value of Cronbach’s alpha (or similar measure of reliability) for each assessment overall and for each component of each assessment? By a component is meant here any aspect of an assessment that would be reported separately to the teacher who has administered the assessment
     
  2. What evidence was used to estimate the measure of reliability presented in answer to (1)? The answer should specify the number of pupils on which the assessment was trialled, and the conditions under which the trial was done.
     
  3. How was the validity of the assessment evaluated in relation to Curriculum for Excellence experiences and outcomes? There are two aspects of this question: a. What theoretical judgements were made of whether and to what extent the assessments reflect Curriculum for Excellence experiences and outcomes? Who was involved in making these theoretical judgement? In particular, how many recently experienced teachers in Scottish schools were used in making these theoretical judgements? b. What empirical testing was done to compare the conclusion of the assessments with other sources of information about pupils’ achievement in terms of Curriculum for Excellence experiences and outcomes?
     
  4. How were the assessments pre-tested? Did the pre-testing use current pupils in Scottish schools? Did the pre-testing go beyond the estimate of reliability and validity noted in (1), (2) and (3) above? What evidence was recorded of pupils’ response to the assessments – for example, whether they enjoyed doing them, whether they found them stressful, and whether they believed that the assessments would help their learning.
     
  5. Did any pre-testing gather the opinions of teachers in Scottish schools about the routine use of the assessments as formative assessment?
     
  6. How was the fairness of the assessments measured? In particular, how was it measured in relation to pupils recorded as having additional support needs, and in relation to pupils whose main home language was not English?' ‘Assessment’ here refers to any test included within the suite of tests that come under the heading ‘Scottish National Standardised Assessments’.
 
Response
 
Response to to Q1

 The Cronbach’s Alpha analyses of each component of the Scottish National Standardised Assessments are as follows:

Form

Reliability

P1 numeracy

0.840

P1 literacy

0.849

P4 numeracy

0.868

P4 reading

0.880

P4 writing

0.882

P7 numeracy

0.889

P7 reading

0.860

P7 writing

0.820

S3 numeracy

0.880

S3 reading

0.887

S3 writing

0.780

Typically, reliabilities above 0.75 are considered satisfactory and above 0.80 as excellent.

Response to to Q2

 The figures at Q1 were derived from two norming trials, in November 2017 and March 2018, a long scale equating study in February 2018 and the responses given by all children and young people in years P1, P4, P7 and S3 during the presentation of the SNSA during session 2017 – 2018.

The norming studies had participation numbers of:

  • November 2017 – 7,438 assessments
  • March 2018 – 7,307 assessments

The equating study had participation numbers of:

  • February 2018 – 16,052 assessments

Norming studies comprised current children and young people in years P1, P4, P7 and S3. The equating study comprised participants from year groups P2, P3, P5, P6, S1 and S2.

During the course of session 2017-2018, 578,000 assessments were undertaken across all year groups P1, P4, P7 and S3. By year group:-

  •     P1 108,325
  •     P4 167,180
  • P7 161,533
  •     S3 141,347
  • Together with the norming and equating studies, these responses were used to inform calibration of the standardised long scale of the SNSA and Cronbach’s Alpha figures in the table at (1).

    Conditions for the trials and norming and equating studies were designed to replicate the normal classroom environment within which the SNSA, given their purpose to provide formative, diagnostic information to teachers to inform next steps in learning, are designed specifically.

    Response to to Q3

     

  • All questions within the SNSA were assessed and approved for inclusion by experienced professionals within Education Scotland to ensure quality of questions and range of types to align with Curriculum for Excellence experiences and outcomes and the benchmarks. In addition, items were assessed for cultural appropriateness and to ensure questions in each component of the assessment achieved a breadth of coverage and a progression of concepts and skills in curricular areas, in accordance with Curriculum for Excellence.
  •  

  • The SNSA are designed to provide formative, diagnostic information to teachers to inform next steps in learning and teaching in aspects of literacy and numeracy from Curriculum for Excellence. Achievement of a level is a matter of professional judgement, based on a wide range of assessment evidence, of which the SNSA are one source only.

Response to to Q4

 In the SNSA trials, two hundred responses were sought for each of 28 different assessment forms in the trialling phase, from current children and young people in the target year groups: P1, P4, P7 and S3. The target of 200 responses per item was met for 23 of the 28 assessments. For the remaining five (all at S3), the minimum number of responses was 140, sufficient for indicative interpretation. This response rate was far more positive than responses experienced with similar activities in other locations. By the end of the trialling window, approximately 3900 pupils at a total of 66 schools in five local authorities had participated. In total, 9800 forms – including learner assessment and practice forms, staff forms, and technical readiness assessments – were completed. Over 6500 completed assessments were available for psychometric analysis. The figures above provided evidence in line with or in excess of evidence required to ensure reliability and validity of psychometric assessment.

The SNSA trials in February to March 2017 sought the views of teachers participating on a voluntary basis. During the trails, there was no direct recording of the views of children and young people.

Following the SNSA User Review 2017-18, plans are in place to record the views of children and young people presented with the SNSA during the course of 2018-19.

Response to to Q5

The SNSA were introduced as part of the wider National Improvement Framework, introduced in response to evidence presented by OECD, as well as from other findings. User Assurance Groups, with representation of teachers, head teachers, professional associations, local authority officials, academics and specialists in ASN and accessibility, were consulted on aspects of the design and implementation of the SNSA, in addition to the information gathered from site visits and focus groups of teachers throughout the 2017-2018 session.

Following the SNSA User Review 2017-18, teachers will have the opportunity to record their views of the SNSA through the SNSA platform during the course of 2018-19.

Response to to Q6

Prior to implementation of the SNSA in August 2017, a User Assurance Group for ASN and Accessibility was established to seek professional advice on all aspects of the implementation and design of the SNSA. In addition, site visits to schools providing support to children and young people with additional support needs and English as an additional language provided useful feedback to inform improvements to the customisation and design of questions and their presentation within the assessment. All aspects of the SNSA were assessed against international standards for accessibility. The advice received informed enhancements to the SNSA in advance of launch in August 2017.

 
About FOI
The Scottish Government is committed to publishing all information released in response to Freedom of Information requests. View all FOI responses at http://www.gov.scot/foi-responses

Contact

Please quote the FOI reference
Central Enquiry Unit 
Email: ceu@gov.scot
Phone: 0300 244 4000 

The Scottish Government 
St Andrew's House 
Regent Road 
Edinburgh 
EH1 3DG

Back to top