National Qualifications experience 2020: rapid review

Footnotes

1. We initially expressed, to the government, an interest in conducting analysis of the dataset using the algorithm employed for grading. It was made clear to us that, while the government would make a request of SQA for the data, the request was unlikely to be granted in the short time scales of a rapid review. Subsequently, and due to challenges in resourcing and gaining university ethical clearance for such an analysis within the timeframe, we did not pursue this option. We do, however, believe that such an analysis is necessary to gain a full understanding of the processes undertaken, hence our recommendation 8.

2. https://www.bera.ac.uk/publication/ethical-guidelines-for-educational-research-2018

3. As reported by young people and the organisations representing them.

4. We also note here that SQA provided additional guidance to centres and historical estimate and results data, for the purpose of allowing centres to review at dept level whether they got it about right or that they had a tendency to over or under estimate.

5. SQA provided communications to centres on estimation on the 2^nd April and 20^th April.

6. SQA states that they consulted stakeholders on this, including subject associations.

7. We note that nearly 100 centres are not within LA jurisdiction.

8. There was qualitative input from SQA's subject Principal Assessors, Qualifications Managers and Heads of Service into defining the Starting Point Distributions and reviewing the model outcomes.

9. SQA's position is that 'to include any considerations of socio-economic status into the model and/or seek to validate with Local Authorities, would have made the approach subjective and introduced 'bias' and perceptions of bias into the process'.

10. Although this is based on A-levels we do not have reasons to think that it operates different in Scotland.

11. If any centre had only one or two years' attainment history on a course for which they had entries in 2020, then the historic range for that centre on that grade was extended in each direction, to provide a range of five ventile bands. The additional allowance of two ventiles in each direction is then further applied to this extended ventile range. Centres with no history (i.e. presenting entries for a course for the first time) were therefore awarded the original estimates submitted by their centres (SQA, 2020, p.36)

12. Based on the information provided by head teachers and LAs. A full analysis of the datasets would be needed to confirm this.

13. Based on the information provided by head teachers and LAs. A full analysis of the datasets would be needed to confirm this.

14. Distributions (SPD) to introduce adjustments to the tolerances, based on centre-level constraints. The SPDs were estimated as a proportional national attainment level for each grade on a given course. Although the SQA (2020) Technical Report says that they "sought to take the average of as many recent comparable years of attainment data as was available for the course", it subsequently clarified the SPDs were based on two year (2018 and 2019) averages only for National 5 courses, while the SPDs produced for Higher and Advanced Higher courses were predominantly based on 2019 data (p. 29). The latter means that, if the 2019 national result were particularly low for some courses (which we know has been the case for some Higher courses), that could have affected the acceptable tolerances for these courses and downgraded the centre results more than it should, based on the centre's historical attainment alone.

15. In 2019 for National 5, at least half of class entry sizes were made up of 19 or fewer candidates; for Higher, at least half of class entry sizes were made up of 14 or fewer candidates; and for Advanced Higher, as least half of class entry sizes were made up of four or fewer candidates (SQA 2020).

16. Based on the moderation vs result data provided by teachers.

17. Based on the information provided by head teachers and LAs. A full analysis of the datasets would be needed to confirm this.

18. SQA guidance for centres (Post-Certification Review – Information for centres), released in June and revised in July stated: 'The alternative certification model is based on teachers' and lecturers' estimates, which have been moderated by schools, colleges and SQA. The process may lead to a candidate or group of candidates being certificated with a grade that's different from their estimated grade. To be as fair as possible to candidates, we are providing a post-certification review (appeals) service to allow centres to request a review of the grade awarded for a candidate or a group of candidates.' While this clearly indicates the possible effects of moderation, it conveys a message that PCR is an appeals process rather than an integral stage in the process.

19. Noting that not all centres sit within local authorities)

20. SQA's position is that meant that any appeals process that did not award based on the original centre estimate was contrary to the Ministerial direction on the 11 August 2020.

21. As stated earlier in the report, more analysis of data is needed to explore these emerging patterns.

22 As stated earlier in the report, more analysis of data is needed to explore these emerging patterns.

23. We note here that SQA offers a range of ways in which candidates can make contact including phone, email, Facebook, Twitter, MySQA Sam, Candidate Enquiry Webform. The consistent perceptions of young people interviewed, that SQA communication is confusing, perhaps indicates the need for greater engagement with young people (building on existing work) to co-construct communication channels and promote their wider use.

24. For the 2018/19 school leaver cohort, 24.5% of school leavers' highest qualification was at SCQF Level 5 (Nat 5 level) (source Scottish Government).