Scottish Household Survey 2019: methodology and fieldwork outcomes

6 Survey Response

Summary

• The final number of social survey interviews achieved was 10,577. This represents a response rate of 63 per cent and exceeds the target of 10,450 interviews.
• The SHS response rate has fallen in recent years, having fallen by three percentage points since 1999/2000. The response rate of 63% in 2019 was one percentage point lower than the previous four years and four percentage points lower than the 2014 response rate of 67%.
• The long-term average response rate for 1999-2011 was 67.9%. However, it should be noted that the calculation has changed slightly for 2012 onwards as a portion of the addresses of unknown eligibility are considered to be eligible whereas previously they would all have been classed as ineligible. This calculation change would have led to a lower response rate in years prior to 2012, if it had been calculated on the same basis.
• There was a target of at least 80 completed physical surveys for each local authority along with a target of 3,004 surveys for Scotland.

6.1 Introduction

This section presents the fieldwork outcomes for the sampled addresses.

The final number of social survey interviews achieved was 10,577. This represents a response rate of 63 per cent and exceeds the target of 10,450 interviews. Since the design changes to the SHS were introduced in 2012 the target number of interviews has been met in recent years, namely 2017 to 2019. However, the SHS response rate has fallen by three percentage points since 1999/2000 and the impact of this drop in response rate is discussed in section 6.3. The performance of the physical survey is described in section 6.5.

The SHS response rate has fallen in recent years, having fallen by three percentage points since 1999. The response rate of 63% in 2019 was one percentage point lower than the previous four years and four percentage points lower than the 2014 response rate of 67%. The maintenance of the response rate in a climate of declining response rates on other population surveys is commendable.

Previously the contractual required response rate was 67%. However, in 2017, the SHS steering group agreed to lower the target in the new contract. The group agreed on a 65% rate for the contract specification 2018-2021. The 2018-2021 contract states that "the target response rate for the survey will be at least 65%".

The SHS 2018-2021 contract states that "The household response rate assumption for each Local Authority will be updated annually and set as the mean response rate for each Local Authority over the last three sweeps of the survey for which response rate data are available. Response rate here is the successful completion of the household interview." This is subject to the following conditions: (i) the response rate assumption for any Local Authority will not be below 55%; (ii) the response rate assumption for any Local Authority will not be above 80%; and (iii) the Scotland level response rate will not be below 65%. If the third condition is at risk of being breached then each Local Authority's response rate assumption will be uniformly increased to ensure the Scotland level response is 65% or above for each survey year.

Survey response is an important indicator of survey quality as non-response can introduce bias into survey estimates. Standardised outcome codes (based on an updated version of those published in Lynn et al (2001)^[20]) for survey fieldwork were applied across the SHS, SHeS and SCJS. The outcome codes paper includes guidance on the appropriate categorisation of interview outcomes. This will allow consistent reporting of fieldwork performance and effective comparison between the performance of the surveys.

6.2 Scotland-Level Summary

Table 6.1 shows a detailed breakdown of the SHS response for all sampled addresses for Scotland. The addresses of unknown eligibility have been allocated as eligible and ineligible proportional to the levels of eligibility for the remainder of the sample. This approach provides a conservative estimate of the response rate as it estimates a high proportion of eligible cases amongst the unknown eligibility addresses.

The table shows that the overall household response rate was 62.6 per cent. This is below the long-term (1999/2000 to 2011) average response rate for the SHS of 67.9 per cent, below the 2012 to 2014 rates (67%) and below the 2015 to 2018 rates (64%). The effects of the drop in response rate are discussed in section 6.3. It should be noted that from 2012 the calculation was changed slightly and a portion of the addresses of unknown eligibility are now considered to be eligible, whereas previously they would all have been classed as ineligible.

Table 6.1: Fieldwork outcomes, Scotland

Fieldwork Outcome	Sample	% of issued	% of eligible
Responding households	10,577	57.4%	62.6%
Random adult interview complete	9,776	53.0%	57.9%
Refused
Office refusal	733	4.0%	4.3%
Refusal at introduction/before interview	3,806	20.7%	22.5%
Broken appointment - no re-contact	237	1.3%	1.4%
Total refused	4,776	25.9%	28.3%
Non-contact
No contact with anyone at the address	488	2.6%	2.9%
Contact made at address, but not with target respondent	366	2.0%	2.2%
Total non-contact	854	4.6%	5.1%
Other non-response
Ill at home during field period	31	0.2%	0.2%
Away or in hospital throughout field period	124	0.7%	0.7%
Physically or mentally unable/incompetent	209	1.1%	1.2%
Language barrier	63	0.3%	0.4%
Lost interview	0	0.0%	0.0%
Other non-response (not covered by categories above)	0	0.0%	0.0%
Total other non-response	427	2.3%	2.5%
Unknown eligibility
Inaccessible	0	0.0%
Unable to locate address	57	0.3%
Unknown if occupied, due to non-contact	209	1.1%
Other unknown eligibility	4	0.0%
Total unknown eligibility	270	1.5%
Estimated eligible addresses in set of unknown eligibility addresses	253	1.4%	1.5%
Total eligible addresses	16,887	91.6%	100.0%
Not eligible
Not yet built / under construction	17	0.1%
Demolished/derelict	51	0.3%
Vacant/empty	887	4.8%
Non-residential	241	1.3%
Address occupied but not resident household	266	1.4%
Communal establishment / institution	26	0.1%
Other ineligible	37	0.2%
Estimated ineligible addresses in set of unknown eligibility addresses	17	0.1%
Total not eligible	1,542	8.4%
All issued addresses	18,429	100.0%

6.3 Drop in response rate

The response rate of 63% in 2019 was one percentage point lower than the previous four years and four percentage points lower than the 2014 response rate of 67%. Response rates are shown in the Table 6.2 below.

Table 6.2: Response rates over time

	1999/00	2001/02	2003/04	2005/06	2007/08	2009/10	2011	2012	2013	2014	2015	2016	2017	2018	2019
Response rate	66%	67%	68%	69%	66%	69%	69%	67%	67%	67%	64%	64%	64%	64%	63%
Achieved sample	30,227	30,639	30,822	31,013	27,238	28,404	14,358	10,644	10,652	10,633	10,325	10,470	10,683	10,532	10,577

The long-term average response rate for 1999-2011 was 67.9%. However, it should be noted that the calculation has changed slightly for 2012 as a portion of the addresses of unknown eligibility are considered to be eligible whereas previously they would all have been classed as ineligible. This calculation change would have led to a lower response rate in years prior to 2012, if it had been calculated on the same basis.

Due to a drop in the response rate in 2015, an analysis was conducted to investigate the likely impact of this drop in response rates. This was undertaken by modeling the 2014 sample results to examine what the impact of a 3% drop in response rates would have been had a lower number of interviews been achieved, and comparing this against the 67% full sample results.

Analysis showed that the demographic composition of the sample was largely unchanged (age, gender), with only the most deprived SIMD quintile and other urban areas showing a 1 percentage point drop in their respective shares of the total sample.

Analysis of mean differences in the population estimates from the two samples for a basket of full sample questions from the SHS, and full one third sample questions from the SHCS module, were also undertaken.

This showed that the absolute mean differences for the total population estimates across the different basket of questions within the household and random adult parts of the survey, including the Scotland Performs National Indicators, were very small, at around 0.1 percentage points.

A few sub-group categories had one or two 'maximum' differences in estimates of around 1 percentage points, including age and social and private-rented sector sub-groups. However, these differences are unlikely to be statistically significant due to small subgroup sample sizes.

Sub-national analysis was not considered. It is expected that there would be a greater impact of this lower response rate for Local Authorities and other sub-national geographies.

6.4 Local authority performance

Table 6.3 shows levels of ineligible addresses, response rate and random adult conversion.

Table 6.3: Response rate and eligibility by local authority

		Ineligible addresses		Responding households		Random adult interviews
Local Authority	Sampled addresses	n	%	n	% of eligible	n	% of HH ints
Aberdeen City	677	122	18%	314	55%	304	97%
Aberdeenshire	620	63	10%	363	65%	342	94%
Angus	472	39	8%	251	58%	234	93%
Argyll and Bute	457	89	19%	249	68%	243	98%
Clackmannanshire	372	24	6%	222	64%	213	96%
Dumfries and Galloway	431	43	10%	277	71%	256	92%
Dundee City	411	38	9%	244	65%	219	90%
East Ayrshire	488	31	6%	275	60%	259	94%
East Dunbartonshire	468	13	3%	276	61%	258	93%
East Lothian	431	30	7%	297	74%	273	92%
East Renfrewshire	457	17	4%	259	59%	235	91%
Edinburgh City	1,489	110	7%	871	63%	839	96%
Falkirk	442	19	4%	237	56%	223	94%
Fife	911	68	7%	526	62%	477	91%
Glasgow City	1,987	163	8%	977	54%	853	87%
Highland	648	88	14%	345	62%	329	95%
Inverclyde	435	32	7%	247	61%	236	96%
Midlothian	457	15	3%	291	66%	279	96%
Moray	431	42	10%	256	66%	225	88%
Na h-Eileanan Siar	383	49	13%	278	83%	275	99%
North Ayrshire	481	41	9%	251	57%	233	93%
North Lanarkshire	885	49	6%	522	62%	462	89%
Orkney	367	50	14%	258	81%	246	95%
Perth and Kinross	428	22	5%	237	58%	203	86%
Renfrewshire	478	14	3%	263	57%	249	95%
Scottish Borders	432	42	10%	267	68%	239	90%
Shetland	376	40	11%	244	73%	217	89%
South Ayrshire	449	43	10%	263	65%	256	97%
South Lanarkshire	862	60	7%	480	60%	423	88%
Stirling	376	27	7%	240	69%	212	88%
West Dunbartonshire	426	22	5%	243	60%	236	97%
West Lothian	402	20	5%	254	66%	228	90%
Scotland	18,429	1,525	8%	10,577	63%	9,776	92%

Argyll and Bute, Aberdeen City, Orkney, Highland, Na h-Eileanan Siar, Shetland and Aberdeenshire^[21] were the areas where the highest levels of ineligible addresses were recorded. For Argyll and Bute and Na h-Eileanan Siar, high levels of ineligible addresses were expected as both areas contain a high number of holiday and second homes which are not eligible for the survey. Expected levels of ineligible addresses for Highland, Orkney and Shetland were all above 10%.

The two lowest household response rates in 2019 were in Glasgow City (54%) and Aberdeen City (55%). In addition, seven other local authorities had a response rate under 60 per cent in 2019.

The conversion from household interview to random adult completion was
92% in 2019 (this was 92% in 2016 to 2018, 91% in 2015, 92% in 2014 and 93% in 2012 and 2013).

6.5 Monitoring and reducing the respondent burden

The Code of Practice for Statistics^[22] states that "Statistics producers should be transparent in their approach to monitoring and reducing the burden on those providing their information, and on those involved in collecting, recording and supplying data." Furthermore it states that "The burden imposed should be proportionate to the benefits arising from the use of the statistics."

The following steps are/have been taken to reduce respondent burden.

• Addresses selected for any of the surveys (SHS, SHeS, SCJS) are removed from the sample frame for a minimum of 4 years so that they cannot be re-sampled for another survey.
• Samples sizes are no greater than required to obtain robust local authority estimates.
• The size of the SHS questionnaire was not allowed to increase following the recent consultation^[23] on its contents. Where there was a strong policy need for new questions, the space/time required for these new questions was created by dropping existing questions or moving them to a biennial basis.
• The SHS social interview is carried out using Computer Aided Personal Interviewing (CAPI). Routing is built into the CAPI script which ensures that respondents are only asked questions that are relevant to them, e.g. respondents are only asked questions on mortgage repayments if they have a mortgage.
• Interviewer briefings are held annually. These events provide interviewers with an opportunity to feedback on improvements that could be made to the questionnaire to assist them conducting interviews and reduce the burden on respondents.
• Interviewers were asked to choose the logo on the tote bag which was given to respondents as an incentive to complete the survey in 2019.
• The Scottish Household Survey team engages its users in the design of new products and in the steering and evaluation of the dissemination of the survey results. The Scottish Household Survey run an annual user day and we normally have around 70-80 attendees every year. We advertise this on the Scotstat email distribution list and through twitter. In addition, we send out annual report evaluation questionnaires to our users to find out views of our publications.
• For large-scale new developments such as the change away from our Excel tables to our new web-based data explorer, we have collected views user through quantitative questionnaires – but also through face-to-face testing and meetings with local authority staff.

6.6 Physical survey

As described in section 2, just under half of the SHS sample was assigned to the physical survey module. For completion of the physical survey, respondents had to agree to make an appointment for a surveyor to make a follow-up visit and to complete the appointment. Table 6.4 shows the number of households assigned to the physical module which responded to the main SHS and the conversion to completion of the physical survey.

There was a target of at least 80 completed physical surveys for each local authority along with a target of 3,004 surveys for Scotland. The result of continuing lower than estimated levels of conversion from household survey to physical survey, meant that there were fewer than 80 physical survey responses for ten of the local authorities in 2019^[24]. However, the decision to 'open up' an additional stream to the physical survey gave a lower shortfall on the minimum target of 80 per local authority than in previous years (11 local authorities in 2018, 9 in 2017, 16 in 2016, 17 in 2015, 24 in 2014, 21 in 2013 and 20 in 2012). The 2,997 surveys achieved for Scotland in 2019 was 7 fewer than the target of 3,004.

Table 6.4: Conversion to full physical survey

Local Authority	Conversion rate from household interview to physical survey	Physical survey complete
Aberdeen City	63.1%	77
Aberdeenshire	59.0%	82
Angus	73.2%	90
Argyll and Bute	60.3%	70
Clackmannanshire	71.0%	71
Dumfries and Galloway	72.0%	90
Dundee City	61.6%	77
East Ayrshire	52.6%	70
East Dunbartonshire	59.4%	82
East Lothian	57.5%	77
East Renfrewshire	62.7%	84
Edinburgh City	57.3%	208
Falkirk	70.3%	90
Fife	59.1%	140
Glasgow City	55.1%	225
Highland	64.7%	86
Inverclyde	53.4%	71
Midlothian	59.2%	90
Moray	66.1%	82
Na h-Eileanan Siar	61.8%	89
North Ayrshire	71.7%	86
North Lanarkshire	51.8%	103
Orkney	82.6%	100
Perth and Kinross	72.1%	88
Renfrewshire	48.2%	54
Scottish Borders	54.2%	77
Shetland	76.6%	85
South Ayrshire	68.9%	82
South Lanarkshire	52.6%	110
Stirling	64.9%	74
West Dunbartonshire	65.1%	82
West Lothian	76.6%	105
Scotland	61.9%	2,997