Long term survey strategy: mixed mode research report
Findings from research exploring mixed mode survey designs in the context of the Scottish Government’s general population surveys. The report details information on key issues, potential mitigations and remaining trade-offs, and includes 21 case studies on relevant surveys.
Footnotes
1. https://www.gov.scot/groups/equality-data-improvement-programme-edip-group/
2. See Office for Statistics Regulation website for an explanation of these designations.
3. An indicator of level of exposure to nicotine, from tobacco.
4. The Scottish Government conducts a wide range of surveys. Other major Scottish studies include Growing Up in Scotland (as noted above) and the Scottish Social Attitudes Survey, which the Scottish Government has commissioned modules on. However, the three general population surveys outlined above have been at the core of the Long Term Survey Strategy to date.
5. Scottish Government LTSS, 2018-2022. Office of the Chief Statistician, Scottish Government
6. The one major exception to face-to-face data collection pre-2020 was in 2004, when the then Scottish Crime and Victimisation Survey moved to telephone interviewing. However, after a review of the data questioned the robustness of the new approach, the survey returned to face-to-face interviewing. See Hope, S. (2005); SCVS: Calibration Exercise Report, Edinburgh, Scottish Government.
7. See Scottish Government Coronavirus (Covid-19): Scottish Victimisation Telephone survey 2020 webpages.
9. See Scottish Household Survey 2020: methodology and impact of change in mode report.
10. See The Scottish Health Survey 2021 – volume 2: technical report.
11. See ONS blog on Designing the future of ONS Surveys for more information on this
12. Scottish Household Survey 2022: key findings - Internet
13. Of the UK surveys covered, seven – British Social Attitudes, the British Election Survey, the European Social Survey, the European Health Interview Survey, Natsal, the Transformed Labour Force Survey, and Understanding Society – include Scotland while the rest are specific to other parts of the UK.
14. For example, during the pandemic, when all SHS fieldwork had to be undertaken remotely, respondents were offered the choice of telephone and video interviewing using one-way Microsoft Teams, so that the respondent could see the interviewer, but the interviewer could not see them. However, there was reluctance from both interviewers and respondents to undertake interviews by video and uptake was low. That said, there are examples where take-up of video interviews has been higher – for example, NCDS-65 obtained 30% of all responses by video interview (2020-23) (Durant et al, 2024).
15. For instance, data from the 2021-22 Active Lives Survey, which uses a push-to-web methodology with web and paper modes, found that most responses (71%) were by web, and of these almost half (44%) were completed on smartphones, equating to 31% of all responses.
16. Ipsos has produced a best practice guide on implementing a ‘mobile first’ approach.
17. See Scottish Health Survey 2022 – volume 2: technical report
18. As described in the case study in Appendix A, the NTS is in the process of moving from a paper to a web-based travel diary, however.
19. T-ACASI involves questions being asked by a digitised-voice, or voice actor, and respondents answering by pressing the appropriate keys on their touch-tone telephone. This means the telephone interviewer does not ask the questions, or hear or see the answers. However, it is more technologically challenging to administer than CASI in a face-to-face interview, requiring the integration of separate technology, more effort in explaining to respondents how to use it, and potentially more risk in terms of respondents making mistakes or ringing off in error. This may explain its relatively infrequent use. See Villarroel et al (2008) for an example of its use in practice.
20. Random digit dialing (RDD) involves using software to randomly generate phone numbers. Landlines can be generated using the first part of area codes, so that they can be targeted at specific geographies. This is more difficult with mobile numbers; although new techniques have emerged that allow randomly generated mobile numbers to be ‘pinged’ to estimate where they were last used, this is much less accurate.
21. Consumer and Demographic data, collated by CACI.
22. For a discussion of challenges applying TSE, see for example, Robert M. Groves, Lars Lyberg, Total Survey Error: Past, Present, and Future, Public Opinion Quarterly, Volume 74, Issue 5, 2010, Pages 849–879
23. See Mitra, R et al (2021) Knowledge Series 1 Population Registers: Definitions and Conceptual framework
24. Technical specification for SAD, accessed from nrscotland.gov.uk August 2024
25. OfCom’s 2023 Technology tracker found that 36% of people in Scotland said they did not have a landline. Across the UK, over 50% of those under 55 said they did not have a landline.
26. Developing an ONS Population spine
27. See in particular projects led by Smith and Nicolaas, as detailed on the Survey Futures website.
28. Martin, C. (2022) Scottish Household Survey: methodology and impact of change in mode. Scottish Government web-based report
29. See, for example, section 2.3.1 in Scottish Crime and Justice Survey 2021/22 Technical report
30. See section 2.3.2 in Scottish Crime and Justice Survey 2021/22 Technical report
31. See Scottish Household Survey 2022: methodology and fieldwork outcomes
32. See The Scottish Health Survey 2022 edition: Volume 2 – Technical report
33. There are some variants to this approach. Surveys can simply allow any two adults to complete the survey (self-selection) or can attempt to select a random two. In addition, they can either select both upfront, or can ask one person to complete the survey first, and only after the household grid is completed make a request for a second person to complete the survey.
34. The American Association of Public Opinion Research (AAPOR) has developed guidance on how to produce comparable response rates, as far as possible, across surveys.
35. See Labour Force Survey performance and quality monitoring report: January to March 2019
36. See reports on the development work on the NHS surveys website
37. For some measures, it was difficult to unpack whether changes reflected change in mode/design or genuine changes relating to the pandemic – for example, the proportion of people feeling lonely, able to rely on neighbours, attending cultural activities, and visiting the outdoors all changed.
38. Martin, C. (2022) Scottish Household Survey: methodology and impact of change in mode. Scottish Government web-based report
39. In 2019, 51% of adults in ‘co-operating households’ that took part in at least part of the main interview also took part in a nurse visit. In 2021, the nurse visit response rate among ‘co-operating households’ fell to 24%. See EHS method reports for 2019 and 2021
40. Longitudinal surveys, like Understanding Society, can use information the characteristics of households and individuals, as well as information about who was more or less likely to respond by particular modes, to develop much more sophisticated targeting strategies to combat nonresponse and bias.
41. Note that the National Travel Survey, which is in the process of moving from a paper-based to a web-based travel diary, is likely to provide further evidence on this in future. See case study in Annex A.
42. See for example Bhatia, A et al (2022) ‘Remote methods for research on violence against women and children: lessons and challenges from research during the COVID-19 pandemic’ in BMJ Global Health
43. See reports on development and testing on the NHS surveys webpages
44. See SCJS FAQs – the mean current length is 42.5 minutes (based on the contractor’s monitoring data)
45. At the time of writing, this was the topic of a current project under the ESRC Survey Futures initiative.
46. Labour Market Transformation: update on progress and plans: July 2024
47. See discussion of the impacts of reweighting on LFS key indicators on the ONS website.
48. See the Socia Research Association Explore the Data: Food and You 2 blog
49. See description of ESS on the Eurostat website and ONS Statistical Quality Improvement Strategy - Office for National Statistics
50. See Social Research Association Future of random probability surveys roundtable session
51. The system is called ASPIRE (A System for Product Improvement, Review and Evaluation) and was developed with Paul Biemer (RTI) and Dennis Trewin (Former Australian Statistician). For further information, see Biemer, P.B., Trewin, D. Bergdhal, H. & Xie, Y. (2017).
52. The Scottish census cost £63.5 million for 2011 compared with an estimated £138.6 million for 2022 (as documented in FOI request 202200291286, accessed June 2024). The England and Wales 2011 census had cost £478 million, compared with £207 million for the 2001 census See Census 2021 response to FOI requests on Cost, prosecutions and spoilt returns for the 2001 and 2021 Censuses
53. See Charman, C et al (2024) The post-pandemic role of face-to-face fieldworkers, NatCen
54. See FOI response Costs of the Transformed Labour Force Survey
55. See presentation to the Royal Statistical Society conference 2022
56. Scotland’s circular economy and waste route map to 2030 – consultation (Jan 2024)
57. See for example Shulte Nordholt, E (2018) The usability of administrative data for register-based censuses and Alleva et al (2021) Measuring the accuracy of aggregates computed from a statistical register
58. Using the same contact strategy, the Inpatient survey, where patients are generally older, 61% took part online in a sequential push-to-web design, compared with 84% on the Maternity survey, which has a much younger service user profile. See CQC Cross Programme Development Work and Pilot results 2020.
59. See Participation Survey 2022 to 2023: Annual technical report
60. See Scotland’s Census 2022 Quality Assurance report
61. See for example, discussion of the fact that only four-fifths of respondents who had experienced sexual assault had reported this to the police on the most recent occasion. Scottish Crime and Justice Survey 2019/20: Main findings
62. AAPOR 1 definition: 47.4% in 2019, 45.8% in 2017, and 55.6% in 2015.
65. The NatCen REMODEL approach – see section 4 on BSA Transformation programme
66. A small supplementary sample of high-income parents (who are undercovered by the Child Benefit register) from those who have completed the Family Resources Survey and consented to recontact, is also issued.
67. See User guide to crime statistics for England and Wales during the pandemic on the ONS website.
68. See discussion of comparability between the telephone operated CSEW and the face-to-face CSEW on the ONS website.
69. See discussion on ONS website
70. See Re-design of CSEW core questions for online collection, on the ONS webpages
71. See Research on Transforming the Crime Survey for England and Wales on the Verian Group webpages
72. See Consultation response: redesign of the CSEW (2022). ONS.
73. See CPS Design and Methodology Technical Paper
74. See Current Population Survey 2023 Modernization Efforts
75. Statistics Netherlands webpage for the CVS
76. For further details, see the UK Government’s EHS webpages
77. See 2022-23 Headline report technical notes and glossary
78. See Food and You Review Report (Paper 3.5) by Professor George Gaskell (2019)
79. See Data collection transformation on the ONS webpages
80. Ultimately, the sample size increased from c. 3,000 interviews every other year, to c. 4,000 interviews twice a year, which on a pro-rata basis equates to an increase of a factor of 5.3.
81. Office for National Statistics. (n.d.). Health and social care statistical outputs. Retrieved August 26, 2024 from Health and social care statistical outputs - Office for National Statistics - Citizen Space (ons.gov.uk).
82. NHS Digital. (2021). The Health survey for England 2020/2021 Feasibility Study. NHS England.
83. NHS Digital. (2022). Health Survey for England 2021 Methods. NHS England.
84. NHS England. (2024). Health Survey for England 2022 Methods. NHS England.
85. NHS Digital. (2020). Health Survey for England 2019 Methods. NHS England.
86. National Survey for Wales 2022-23 Technical Report, p.40.
87. Review of Options for the National Survey for Wales
88. Based on discussions with the commissioning team for the NSW.
89. ‘Fully productive’ response rate – i.e. the whole household participates and returns the travel diary.
90. See National Travel Survey: 2020 technical report
91. See National Travel Survey: 2021 technical report
92. National Travel Survey: 2022 technical report
93. The 2023 wave of the NIHCS is currently undergoing QA and will be published later this year.
94. Northern Ireland House Condition Survey 2016: Main Report
96. DCMS, Participation survey webpage
97. For further information, see Guidance on comparability between Taking Part and the Participation Survey
98. AddressBase Premium | Data Products | OS (ordnancesurvey.co.uk)
99. See Labour market transformation – update on progress and plans: July 2024 on the ONS website.
100. The Understanding Society website includes detailed documentation and technical information on its design at every wave
101. The detail provided here is at a very summary level – for more detail, see the Study Overview pages and associated documentation and technical reports.
102. Full details of these are expected to be published later in 2024.
103. Jackle, A, Lynn, P, Burton, J (2015) Going online with a face-to-face household panel: effects of a mixed mode design on item and unit non-response
Contact
Email: sscq@gov.scot