Validity and reliability of measurements of the National Standardised Assessments: FOI release

 The Cronbach’s Alpha analyses of each component of the Scottish National Standardised Assessments are as follows:

Typically, reliabilities above 0.75 are considered satisfactory and above 0.80 as excellent.

Response to to Q2

 The figures at Q1 were derived from two norming trials, in November 2017 and March 2018, a long scale equating study in February 2018 and the responses given by all children and young people in years P1, P4, P7 and S3 during the presentation of the SNSA during session 2017 – 2018.

The norming studies had participation numbers of:

• November 2017 – 7,438 assessments
• March 2018 – 7,307 assessments

The equating study had participation numbers of:

• February 2018 – 16,052 assessments

Norming studies comprised current children and young people in years P1, P4, P7 and S3. The equating study comprised participants from year groups P2, P3, P5, P6, S1 and S2.

During the course of session 2017-2018, 578,000 assessments were undertaken across all year groups P1, P4, P7 and S3. By year group:-

•     P1 108,325
•     P4 167,180
• P7 161,533
•     S3 141,347

• Together with the norming and equating studies, these responses were used to inform calibration of the standardised long scale of the SNSA and Cronbach’s Alpha figures in the table at (1).

  Conditions for the trials and norming and equating studies were designed to replicate the normal classroom environment within which the SNSA, given their purpose to provide formative, diagnostic information to teachers to inform next steps in learning, are designed specifically.

  Response to to Q3

   

• All questions within the SNSA were assessed and approved for inclusion by experienced professionals within Education Scotland to ensure quality of questions and range of types to align with Curriculum for Excellence experiences and outcomes and the benchmarks. In addition, items were assessed for cultural appropriateness and to ensure questions in each component of the assessment achieved a breadth of coverage and a progression of concepts and skills in curricular areas, in accordance with Curriculum for Excellence.

•  

• The SNSA are designed to provide formative, diagnostic information to teachers to inform next steps in learning and teaching in aspects of literacy and numeracy from Curriculum for Excellence. Achievement of a level is a matter of professional judgement, based on a wide range of assessment evidence, of which the SNSA are one source only.

Response to to Q4

 In the SNSA trials, two hundred responses were sought for each of 28 different assessment forms in the trialling phase, from current children and young people in the target year groups: P1, P4, P7 and S3. The target of 200 responses per item was met for 23 of the 28 assessments. For the remaining five (all at S3), the minimum number of responses was 140, sufficient for indicative interpretation. This response rate was far more positive than responses experienced with similar activities in other locations. By the end of the trialling window, approximately 3900 pupils at a total of 66 schools in five local authorities had participated. In total, 9800 forms – including learner assessment and practice forms, staff forms, and technical readiness assessments – were completed. Over 6500 completed assessments were available for psychometric analysis. The figures above provided evidence in line with or in excess of evidence required to ensure reliability and validity of psychometric assessment.

The SNSA trials in February to March 2017 sought the views of teachers participating on a voluntary basis. During the trails, there was no direct recording of the views of children and young people.

Following the SNSA User Review 2017-18, plans are in place to record the views of children and young people presented with the SNSA during the course of 2018-19.

Response to to Q5

The SNSA were introduced as part of the wider National Improvement Framework, introduced in response to evidence presented by OECD, as well as from other findings. User Assurance Groups, with representation of teachers, head teachers, professional associations, local authority officials, academics and specialists in ASN and accessibility, were consulted on aspects of the design and implementation of the SNSA, in addition to the information gathered from site visits and focus groups of teachers throughout the 2017-2018 session.

Following the SNSA User Review 2017-18, teachers will have the opportunity to record their views of the SNSA through the SNSA platform during the course of 2018-19.

Response to to Q6

Prior to implementation of the SNSA in August 2017, a User Assurance Group for ASN and Accessibility was established to seek professional advice on all aspects of the implementation and design of the SNSA. In addition, site visits to schools providing support to children and young people with additional support needs and English as an additional language provided useful feedback to inform improvements to the customisation and design of questions and their presentation within the assessment. All aspects of the SNSA were assessed against international standards for accessibility. The advice received informed enhancements to the SNSA in advance of launch in August 2017.