Annex A: Methodology - Rapid evidence review
Rapid evidence review methodology (also referred to as Rapid Evidence Assessment) was utilised to review a wide range of existing and current evidence. A rapid evidence review provides a more structured and rigorous search and quality assessment of the evidence than a literature review but is not as exhaustive as a systematic evidence review.
Rapid evidence reviews are most often employed where rapid access is needed to high quality evidence to inform policy, decisions and planning on emergent issues. Given the imperative to synthesise the large body of evidence available in relation to the impact of the COVID-19 pandemic on the educational experiences and attainment of children and young people affected by poverty, to be of utility to policy makers and the education sector, a rapid evidence review was considered the most appropriate methodology.
Rapid evidence reviews employ a specific methodology that aims to identify the most relevant studies on a specific topic as comprehensively as possible. A key inquiry question (and sub questions – set out in the table further below) are agreed defining the search terms and methods for extracting evidence.
The selection of studies is based on explicit criteria. An element of quality appraisal, considering the methodological quality of the studies is also assessed. A synthesis of what the evidence indicates is produced usually in the form of a narrative report. The process of developing an evidence review generally follows the key stages set out below:
1. Develop inquiry question – What does this rapid evidence review answer?
2. Inclusion Criteria – Which studies will be taken into account?
3. Search Strategy – How were the studies sought?
4. Selection process – How were the studies selected?
5. Critical Appraisal – How was the quality of the studies determined?
6. Results – Definitions and thematic summaries
7. Synthesis of main findings
Equity Audit Stage 1 – Rapid Evidence Review - Methodology
Key Inquiry question:
1. What does the existing research and wider evidence suggest has been the impact of the COVID-19 pandemic on the educational experiences and attainment of children and young people, and in particular those children and young people affected by poverty.
1. What has been the impact of the COVID-19 pandemic on educational experiences and attainment – for all children and young people and in particular for children and young people affected by poverty?
2. What evidence is there from children and young people’s perspective/voice and in particular on the voice of children and young people whose educational experiences and attainment are affected by the poverty-related attainment gap, on the impacts of school building closures and move to remote learning?
3. What policy and practice responses have education systems developed during the COVID-19 pandemic specifically to address children and young people whose educational experiences and attainment are affected by the poverty-related attainment gap and what does the emerging research/evidence base suggest has been the impact of policy and practice responses?
4. What factors are emerging which are associated with mitigating the impact of COVID-19 on children and young whose educational experiences and attainment are affected by the poverty-related attainment gap?
1. Material published in English between 23/3/20 to 1/9/20 covering Scotland/devolved nations/UK-wide /Europe/International
2. Material focused on children and young people aged 5 up to age 18 in a school setting (early years is out of scope)
3. Material focused on educational experiences, outcomes with focus on poverty disadvantage, interaction of poverty disadvantage with ethnicity, ASN/SEN
4. Material included- surveys and other primary research, systematic review, think pieces, narrative review, articles
1. Material not published in English
2. Material related to higher or further education
3. Material with a clinical focus
4. Articles that are opinion pieces, editorials or letters
5. Studies that have a pre-COVID-19 focus on impact of school building closures (Out of scope however some material will be used as context setting)
Three key methods were employed to gather relevant evidence:
1) Systematic searches conducted on our behalf by Scottish Government Library
2) Evidence gathered through stakeholder engagement exercise with:
- members of CERG, and
- with Research Strategy for Education in Scotland National Advisory Group (NAG) and Academic Reference Group (ARG)
3) Ad hoc evidence gathered by Equity Audit Phase 1 Short Life Working Group members through ongoing research engagement, website monitoring, newsfeeds and other updates.
Education 5 – 18 years
School (building) closures
Pupil wellbeing/mental health
Teacher wellbeing/mental health
We focused on the quality and rigour of the evidence and in addition we identified the difference in source material between research studies and evidence collection. We also took into consideration of a significant number of media articles and think pieces. Editorials, articles promoting a specific agenda or viewpoint unsubstantiated by evidence were discounted. Assessing the broad scope of material available was crucial in gaining a clear picture of what the evidence is telling us. In light of this we considered other rapid evidence reviews and evidence collected by key stakeholders such as third sector organisations and charitable bodies.
Results, Definitions and thematic summaries
All ‘in scope’ relevant references identified through the literature searches were added to the spreadsheet. Initial thematic coding was developed to code material. Working group members created thematic summaries for the majority of themes.
- Policy approaches
- Global policy responses – generic
- Global policy responses – specific to disadvantage/poverty/deprivation
- Changing economic circumstances
- Remote learning
- Blended learning
- Recovery – re-opening schools
- ‘Learning loss’ and attainment
- Learning experiences
Voice (cross cutting theme)
- Child/pupil voice
- Parent voice
- Teacher voice
- Parental involvement/engagement
- Availability of resources in the home
- Digital divide
- Food poverty
- Vulnerable young people
- Care-experienced children and young people
- Children and young people with caring responsibilities
- New vulnerabilities
Mental health and wellbeing
Synthesis of key findings
Thematic summaries were used develop a narrative report which highlighted key findings from the evidence, structured according to the inquiry questions.