Long term survey strategy: mixed mode research report

10. Financial and resource implications

Introduction

A key factor in considering potential options for keeping, changing, or mixing modes on the Scottish Government’s general population surveys is, of course, the financial and resource implications. Public sector budgets in Scotland, as elsewhere in the UK, are under significant pressure, and survey research is not immune from that pressure.

The costs of delivering high quality surveys are, in general, increasing. This reflects both general inflationary pressures (which impact on fieldwork and other pay-related elements of survey costs, as well as elements like postage and incentives), and decreasing response rates (discussed in chapter 5), which mean that more contacts are required to achieve the same response, and each contact costs more. While inflationary pressures have affected all parts of the economy, they have had a particular impact on the cost of face-to-face fieldwork, given increases in the National Living Wage over the past decade or so (Charman et al, 2024). In the UK, the costs of delivering the most recent censuses were roughly double those incurred a decade earlier.^[52] While surveys are different from censuses, many of the elements driving up their costs are similar (e.g. fieldwork costs, postage costs, the need to invest more to get the same response, even though censuses are compulsory). In reflecting on the current financial climate for surveys, the Office for Statistics Regulation report “The State of the Statistical System 2022/23” describes a “backdrop of financial, resourcing and capability pressures” in which data collectors are working to collect and analyse larger amounts of more complex data (OSR, 2023).

This chapter starts by considering what constitutes ‘value for money’ in the context of survey research and how this might be assessed, before discussing the potential financial and resource implications of changing or mixing modes for the three flagship Scottish Government general population surveys, in the short, medium and longer-term.

Assessing the value for money of survey research

Assessing the value for money to the public purse of survey research is not necessarily as simple as ‘cheaper = better’, since any assessment will almost certainly involve a quality-cost trade-off. This means that weighing up the ‘value for money’ implications of any change in mode on the three Scottish Government surveys will need to involve an assessment of the likely impact for all the elements of survey quality discussed in the preceding chapters of this report and weighing this against any additional priorities and available budgets.

This was reflected in comments from stakeholders interviewed for this study, who saw a close link between value for money and quality. There was a strong view that if a change in mode reduced the quality of the data and resulted in a loss of trust in the findings, then any associated cost savings would be a false economy in terms of real value for money. This was particularly discussed in the context of representativeness: if the surveys were not representative, this would immediately call into question the usefulness of their findings and the ability to draw conclusions from them that can inform policy. As such, any change to the mode design of the surveys would need to consider the issues discussed in chapters 4 and 5 particularly carefully.

Sample size – which is another determinant of accuracy and, in turn, survey quality – is another consideration here. In other words – how many interviews, of the desired quality, it is possible to achieve for a given budget. Survey costs were often discussed by Scottish Government survey stakeholders alongside sample size, with considerable interest expressed in whether mixed mode designs might be able to deliver bigger sample sizes within the same budget. This reflected pressure from stakeholders both within and outwith the Scottish Government to increase the sample size on all three surveys to enhance the precision of sub-group analysis, particularly with respect to equalities groups and specific local geographies (local authority for SHS and Health Board for SHeS), but also for subgroups key to particular policy areas, such as victims (SCJS) and people with specific health conditions (SHeS).

Cost implications of changing or mixing modes

What drives survey costs on different modes?

The biggest variation in costs by mode relates to interviewer costs. Interviewer costs account for a very high percentage of total survey costs on both face-to-face and telephone surveys, but interviewers are not required for web or paper surveys. On face-to-face surveys, travel costs and expenses also need to be factored in. Meanwhile, postal and printing costs will be substantially higher for paper surveys than for other modes, and paper surveys have additional costs relating to scanning and data entry for free text questions. However, unlike other modes, paper surveys will not involve scripting costs. Overall then, the lowest costs tend to be associated with the self-complete modes (web and paper), with telephone costs higher and face-to-face the most expensive.

However, the actual costs of any given survey will also be driven by a range of other decisions, including:

• (All) Sample size – It is important to consider that all surveys have fixed costs and variable costs. The cost of developing, scripting and testing a questionnaire is a fixed cost, while the cost of mailing survey invites, or conducting interviews is variable and depends on the sample size. This implies that the relative cost advantages of cheaper modes are greater the larger the sample size. For a survey with a modest sample size, differences in the unit cost of an interview are likely to be swamped by the large fixed cost of designing and setting up the survey, management, analysis and reporting, and so on.
• (All) Reminder and incentivisation strategies
• (All) Questionnaire length – although the impact of this is more pronounced for interviewer-administered modes, a longer web or paper survey will still cost more than a shorter instrument to develop, implement, and process (and may require higher incentives than an equivalent face-to-face survey)
• (All) Extent of development, testing, and piloting (both short-term and longer-term ongoing development, testing and piloting plans)
• (All) Complexity of other elements, such as sample management, data processing, and weighting
• (For face-to-face and telephone) Reissue strategy (the number of additional contacts required to try and obtain an interview and whether/how this is targeted)
• (For face-to-face) The degree of clustering of the sample (and therefore the distances interviewers travel between interviews)

While overall self-complete modes tend to be associated with lower costs, the scale of the difference between them will also depend on other elements of the design.

Short-term cost implications of changing or mixing modes

While mixed-mode surveys are often discussed in the context of cost savings, delivering a robust and effective mixed-mode survey normally involves investment in design and testing, prior to a successful roll-out. Among expert interviewees who had been, or were currently, involved in moving surveys from face-to-face to another mode or combination of modes, there was a general consensus that changing mode in planned and robust manner results in an increase in short-term costs. More specifically, short-term costs involved in changing or mixing modes are likely to include:

• Questionnaire design and testing – the importance of investing in this process in order to understand how respondents understand questions asked in different modes was underlined by interviewees for this study. The level of time and resource invested in questionnaire design and testing may vary, from more limited cognitive testing of key measures on a new mode, to complete redevelopment of the survey over an extended period. However, for most surveys reviewed for this study, the investment in design has been significant.
• Parallel runs – as discussed in the previous chapter, parallel runs were generally seen as the gold standard for testing and understanding the impact of changes in mode (although it is worth noting a more exceptional view from one expert interviewee that survey teams sometimes spend too much time on parallel runs and would be better focusing limited resources on question testing and piloting). For continuous surveys such as SHS, SHeS, and SCJS, parallel runs by new modes would need to be resourced in addition to the standard costs of continuing to deliver the survey face-to-face. In addition, if both the new mode design and the main survey involve face-to-face or knock-to-nudge elements, this may place an additional strain on limited face-to-face interviewer resources.^[53]
• Experiments to test different incentive and invitation strategies – Across the surveys reviewed for this study, these were generally run alongside or as part of parallel runs to identify those strategies most likely to improve response or mitigate non-response bias in the proposed new design.
• Systems for sample management and data collection – Experts interviewed for this study noted that these might require considerable up-front investment, as case management is more complex when multiple modes are involved, while online data collection platforms and software may also need further development and testing (depending on survey requirements). As fieldwork for the SHS, SHeS, and SCJS are currently delivered via public tender, it would therefore be important that this was included in any tender process. At the same time, all major social research suppliers have experience delivering research via a variety of modes, so may already have the required systems in place.
• Stakeholder engagement – There was a strong recommendation from experts interviewed for this study to communicate with and involve stakeholders in the journey of transitioning surveys as much as possible. This is discussed further in chapter 12, but the resources required to do this effectively should not be underestimated.
• General commissioner and contractor time to plan and manage the transition.

It is also important to factor in the possibility that initial testing might indicate further changes are required, which need to be developed and re-tested before researchers and commissioners can make a robust decision on roll-out of the proposed revised design.

To put this discussion in context, as of March 2024, the cost of ONS’s Transformed Labour Force Survey development, including “survey research, system and technology development, methodological research, printing/materials and postage costs”, as well as the roll-out of the parallel run, has been £24.1 million.^[54] This works out as an average of £8 million per year. By March 2024, the parallel run had been running for over two years, alongside the main survey, to ensure the results from the TLFS were considered robust enough to act as a replacement for the LFS.

While other surveys may not conduct as much additional research, or may decide not to engage in such extensive parallel runs, it is important not to underestimate the costs of survey transformation work in the short term.

Medium-term cost implications of changing or mixing modes

As discussed above, the longer-term cost implications of changing or mixing modes will depend on the exact approach and combination of modes used, and any other changes made at the same time which could increase costs, such as increasing sample sizes or (for non-continuous surveys) increasing frequency.

Moving surveys away from face-to-face, the mode that normally has the highest cost-per-interview rate, can result in a significant saving per interview. Therefore, many mixed-mode approaches will order response modes by cheapest first.

Expert interviews conducted for this study suggest that the greatest cost savings are associated with surveys that move to predominantly self-completion modes (e.g. web first with paper follow-up of non-responders) and make little or no use of face-to-face. One view was that it was possible – even factoring in additional incentive costs – to obtain around three times as many interviews via this approach compared with face-to-face. The Participation Survey has reported achieving this, increasing its sample size from c.10,000 adults to c.33,000 (see case study in Appendix A). The Working Group established by the FSA to consider options for transitioning Food and You away from face-to-face estimated that a push-to-web approach could be expected to reduce the cost per interview from around £130 to around £30 (although it also acknowledge the impact on the time series, the need to shorten the questionnaire, and the potential impact on nonresponse bias – see case study in Appendix A).

In contrast, experts who had been involved in surveys that had either moved to telephone, or which had retained an element of face-to-face follow-up of non-responders, were less likely to report that the long-term savings had been very substantial:

“(it has been) a bit cheaper, not radically. I think sometimes the cost savings of approaches other than face-to-face are a bit overplayed. If you’re doing all you can to get a random sample, then the chasing up work is quite considerable whatever mode you go for. But there is some improvement in costs from having CATI in the mix.” (Expert interviewee 13)

One approach taken on a number of surveys that have moved away from a uni-mode face-to-face design, in order to achieve either cost savings or more interviews for the same budget, is to target budgets for face-to-face follow-up on populations who may be less likely to participate. The Dutch Health Survey uses the population register to target limited funds and resources available for face-to-face follow-up at those with specific demographic characteristics. In the UK, the TLFS uses push-to-web as the first contact mode, and then identifies geographic areas that are less likely to respond (due to local area demographics such as age profile and deprivation) and uses knock-to-nudge approaches to encourage response only in those areas. However, for surveys reliant on local area statistics, there are challenges with identifying target populations at an individual level. For example, while an area may have more people from an ethnic minority background than other areas, the majority of the people living in that area may still be White British.

While these examples indicate that mixing modes can reduce the costs associated with fieldwork, in particular, it is worth noting that some of the other costs of implementing a survey may remain higher in the longer-term after transition to a mixed mode design. In particular:

• Questionnaire design costs for ongoing questionnaire updates may remain be higher as any change to the questionnaire requires testing and implementing in more than one mode.
• Field management costs may remain higher compared with a uni-mode survey because of the additional complexities of case management across more than one mode.
• Data processing costs for mixed mode surveys may be higher in the long-term since, as discussed in chapter 5, weighting data collected by more than one mode can be far more complex (and highly skilled statisticians who can do this work are often in high demand).

What else might impact on survey costs in the longer-term?

In considering the cost implications of any change in design, it is also important to consider that costs are not static. All research modes are seeing increases in costs over time. For example, between March 2022 and March 2024, the cost of a first-class stamp in the UK has increased from 85p to £1.35, which has had a substantial impact on the cost of postal and push-to-web surveys. Response rates are also continuing to decrease over time, across all modes, meaning greater effort is required to achieve the same number of responses. This means that, while there may be significant savings in the medium term, moving mode will not solve all cost concerns for the lifetime of a survey.

In addition, current social trends may make some modes more financially and practically feasible in the future than others. For example, the decrease in use of landlines has more or less ruled out RDD as an approach to developing a robust sample frame for a random probability telephone survey, while the use of call blocking technology on mobiles may make telephone surveys more expensive to conduct even using opt-in approaches. By contrast, as discussed in chapter 1, the proportion of households with internet access is continuing to increase, making online modes more feasible and cost effective for the majority of households.

To address this changing context, most large-scale government-funded surveys will continue to make improvements over time, which may include continuing to explore how to adapt approaches to improve cost savings and/or quality within mixed mode designs. For instance, the GP Patient Survey started as a simultaneous push-to-web survey with paper option in 2006, meaning participants received the paper questionnaire at the same time as the invitation to take part. By conducting randomised control trials on a sub-sample of participants, it was possible to improve this design, while understanding the impact of changes before rolling them out. Using this approach, changes to the contact strategy increased the proportion taking part online from 6.0% in 2017 to 45.5% in 2023, without any impact on trends.^[55] This reduced cost substantially by reducing the amount of return postage and scanning required. On long-running surveys, like SHS, SHeS, and SCJS, it is therefore important to factor in the costs of continued development work to improve quality and/or efficiency into the future costs of survey delivery.

Environmental impact

In addition to financial resources, another factor which survey funders may be concerned about is the environmental resources required to deliver surveys. The Scottish government has set a target to transition to net zero emissions of greenhouse gas and a circular economy, taking into account material consumption and waste management, by 2045.^[56]

All survey modes have some impact on the environment and consideration needs to be given to how to mitigate these (D’Ardenne, 2023). For example:

• Face-to-face interviewers may produce substantial emissions when travelling between interview locations. Clustering sample locations reduces the amount of travel required, which can reduce the environmental impact of the survey, as well as reducing costs.
• Push-to-web surveys require less printing than equivalent sized paper-only surveys, as paper questionnaires can be printed in lower quantities. However, both are likely to involve more printing than face-to-face or telephone approaches. Online surveys also require data storage and hosting, which can result in large emissions due to energy usage.

In addition, there are other aspects of survey research which can have an environmental impact. For instance, e-vouchers require less material consumption than physical vouchers, while gifts require the most consumption and may be more likely to end up in landfill.

To support off-setting the emissions generated from their research, Statistics Austria started offering the option for participants to donate incentive amounts to support renaturation of bogs. A new tool developed by Ipsos UK to estimate the carbon footprint of research studies has also been trialled on the SHS, indicating that 99% of the impact comes from interviewer and surveyor travel.

It is also worth noting that survey findings can play an important role in understanding environmental attitudes and behaviours and informing government policies towards Net Zero. For example, the SHS is used to help monitor sustainable transport modes and provide evidence on the energy efficiency of Scotland’s housing stock. SHS data is used directly for the calculations of GHG emissions in Scotland overall as part of Scotland’s GHG emissions inventory.

More radical options for reallocating resources

The focus of this study was on how changing from one mode to another mode or modes might impact on each of the three main Scottish Government general population surveys. Most of the suggestions above assume that other basic design features and parameters remain the same – in particular, that each survey is conducted separately and that the data they collect is asked directly of respondents. However, it is worth noting that when considering options for obtaining data within limited budgets, a number of more radical suggestions emerged from stakeholder and expert interviews, including:

• Splitting up the surveys into several component parts, with the resultant ‘new’ surveys potentially each using different mode designs. For example, this could involve much shorter ‘core’ face-to-face surveys aimed at retaining key time series, alongside surveys using less expensive modes aimed at collecting data for a larger number of people to enable greater sub-group analysis, and potentially more variation in content over time to meet emerging policy needs. A variation of this approach was adopted on the most recent wave of Natsal (see case study in Appendix A). The survey team was able to reach a larger total sample size within available budgets by moving to conducting only a proportion of their interviews face-to-face, allowing them to maintain key trend data, and then boosting the sample size using a random-probability panel and an online opt-in panel, to examine the experiences of key subgroups in more detail.
• Building in a longitudinal element, to facilitate maximum return from each interview achieved. For example, in order to almost double the sample size in a cost-effective manner, the Crime Survey for England and Wales (CSEW) has introduced a longitudinal element, where participants were contacted for a second wave by telephone. Each published dataset will therefore include around half of participants in their first wave, recruited face-to-face, and half in their second wave, participating by telephone. This was cheaper than increasing the face-to-face element, maintained trends (as face-to-face only elements can be compared over time), allowed for telephone modes to use participant provided telephone numbers, and meant additional analysis looking at repeat victimisation was possible for the first time (for further detail, see case study in Appendix A).
• Combining the Scottish Surveys with UK-wide surveys (like the British Crime Survey)
• Or finding alternative sources for some of the data currently collected via the surveys – including passive data or administrative data.

In relation to this final bullet, there is considerable interest among survey methodologists about the opportunities that smartphones afford for the collection of ‘passive’ data (e.g. Harari et al, 2016; Beuthner et al, 2019; Link et al, 2014). This is data that is collected unobtrusively and in real time by smartphones as a respondent goes about their daily life. Such data has the potential to overcome biases and response tendencies that lead to measurement error, including social desirability bias, satisficing, and telescoping. It can be collected from a range of technologies nascent to smartphones (and smart watches) including accelerometers, microphones, Global Positioning System (GPS) receivers, light sensors, phone and call usage logs, and internet usage logs. These technologies have variously been used to collect data relating to survey respondents’ physical activity, travel patterns, frequency of visiting certain places, sleeping patterns, social interactions, speaking rates, and web browsing and app usage activity (Harari et al, 2016; Revilla et al, 2021).

However, the collection of passive data in survey research is in its early days, and is not without risks which include both issues of non-response (whether from respondents not owning smartphones, respondents refusing to consent to the data being collected, respondents not carrying their smartphones with them or leaving them turned off, and data connectivity issues) and issues of measurement (including technological differences between devices leading to difference in how data is captured, stored and processed). Nonetheless, technological progress over the coming years will likely mean greater opportunities for high quality data to be collected passively, with passive data thus becoming an increasingly relevant part of the social survey landscape.

In relation to administrative data, it is worth noting that stakeholders suggested that making the case for the value for money of Scottish Government surveys was potentially becoming more difficult given that Ministers are also investing heavily in administrative data and data linkage. There was a perceived need for a greater future focus on defining where the surveys are delivering something unique, which cannot be derived from administrative data. The potential interaction between survey and administrative data is discussed in the next chapter of this report.

Summary

Resource considerations do not fit neatly into the ‘issues-mitigations-remaining trade offs’ format used in the summary tables in previous chapters. Rather, the question is how to balance available resources, quality (as determined by all the elements discussed in previous chapters in this report), and sample size (which is also an aspect of quality – but one which is arguably worth separating out given its specific relevance to value for money in the context of different mode designs). There is no straightforward formula for deciding how to balance these. However, decisions should be informed by a clear understanding of overall cost drivers, the short-term implications of changing mode, and medium or longer-term costs. These are summarised in the table below.

Drivers of survey costs

• All modes;
  • Sample size (cost advantages of cheaper modes are greater the bigger the sample size)
  • Reminder and incentivization strategy
  • Questionnaire length
  • Extent of development, piloting and testing
  • Complexity of sample management, data processing and weighting
• Telephone and F2F: Reissue strategy
• F2F: Clustering of the sample.

Short-term resource implications of changing or mixing modes

There was a consensus among experts that changing or mixing modes in a planned and robust manner increases short-term costs.

Short-term costs of changing/mixing modes include:

• Questionnaire design and testing
• Parallel runs
• Experiments to test incentive and invitation strategies
• New/enhanced sample management and data collection systems
• Stakeholder engagement
• General commissioner and contractor time to plan and implement mode change.

Medium-term resource implications of changing or mixing modes

The longer-term resource implications will depend on the exact approach and combination of modes and what other changes are made at the same time (e.g. to frequency, sample size, etc). Given this, it is not possible to make a definitive comparison of the resource implications of different mode designs.

However, costs savings are likely to be greatest where surveys move to predominantly self-completion modes and make little or no use of F2F. Where face-to-face remains a significant element, cost savings tend to be less substantial.

One option to balance this is to target F2F follow-up only on specific demographics (although there are limits to which groups can effectively be targeted in this way).

Other costs that may remain higher longer-term for mixed-mode designs include: questionnaire design; field management; and data processing.

Longer-term cost impacts

Costs are not static. While interviewer administered modes are the most expensive and are likely to see further cost increases (e.g. relating to minimum wage changes), other factors may impact on mixed mode designs too, including:

• Increased postal costs (which can be significant in the context of push-to-web designs, and especially designs where a paper questionnaire may be included for some/all for at least one mailing)
• Call blocking technology may make telephone surveys more expensive in future.

Whatever design is adopted, there is likely to be a need to continue to innovate to improve efficiency as costs change.

There are also environmental resource implications from surveys that require consideration. All survey modes have some environmental impact, including: travel (F2F); printing (push-to-web and paper especially); vouchers (where used).

More radical options for reallocating resources that could be considered alongside mode redesign include:

• Splitting surveys up so that different elements are conducted using different modes
• Building in longitudinal elements to maximise return from each face-to-face interview achieved
• Combining surveys
• Finding alternative sources of data to use instead of/alongside the surveys, including passive data (which is at an early stage in survey research) and administrative data.