6 Survey Response
- The final number of social survey interviews achieved was 10,683, matching the target, a response rate of 64 per cent.
- The SHS response rate has fallen in recent years, having fallen by 4% since 1999. The reduction in fieldwork performance resulted in a response rate of 64% in 2017. This was the same as in the previous year and 3 percentage points lower than the 2014 survey response rate of 67%.
- The long-term average response rate for 1999-2011 was 67.9%. However, it should be noted that the calculation has changed slightly for 2012 as a portion of the addresses of unknown eligibility are considered to be eligible whereas previously they would all have been classed as ineligible. This calculation change would have led to a lower response rate in years prior to 2012, if it had been calculated on the same basis.
- There was a target of at least 80 completed physical surveys for each local authority along with a target of 3,004 surveys for Scotland.
This section presents the fieldwork outcomes for the sampled addresses.
The final number of social survey interviews achieved was 10,683 a response rate of 64 per cent. This was the first year since 2012 design came in that the target number of interviews had been met. The impact of this drop in response rate is discussed in section 6.3. The performance of the physical survey is described in section 6.5.
The SHS response rate has fallen in recent years, having fallen by 4% since 1999. The reduction in fieldwork performance resulted in a response rate of 64% in 2017. This was the same as in the previous year and 3 percentage points lower than the 2014 survey response rate of 67%. The maintenance of the response rate in a climate of declining response rates on other population surveys is commendable.
Survey response is an important indicator of survey quality as non-response can introduce bias into survey estimates. Standardised outcome codes (based on an updated version of those published in Lynn et al (2001)) for survey fieldwork were applied across the SHS, SHeS and SCJS. The outcome codes paper includes guidance on the appropriate categorisation of interview outcomes. This will allow consistent reporting of fieldwork performance and effective comparison between the performance of the surveys.
6.2 Scotland-Level Summary
Table 6.1 shows a detailed breakdown of the SHS response for all sampled addresses for Scotland. The addresses of unknown eligibility have been allocated as eligible and ineligible proportional to the levels of eligibility for the remainder of the sample. This approach provides a conservative estimate of the response rate as it estimates a high proportion of eligible cases amongst the unknown eligibility addresses.
The table shows that the overall household response rate was 63.9 per cent. This is below the long-term (1999/2000 to 2011) average response rate for the SHS of 67.9 per cent and below the 2012 to 2014 rates. The effects of the drop in response rate are discussed in section 6.3. It should be noted that from 2012 the calculation was changed slightly and a portion of the addresses of unknown eligibility are now considered to be eligible, whereas previously they would all have been classed as ineligible.
Table 6.1: Fieldwork outcomes, Scotland
|Fieldwork Outcome||Sample||% of issued||% of eligible|
|Random adult interview complete||9,812||54.4%||58.7%|
|Refusal at introduction/before interview||3,592||19.9%||21.5%|
|Broken appointment - no re-contact||202||1.1%||1.2%|
|No contact with anyone at the address||688||3.8%||4.1%|
|Contact made at address, but not with target respondent||416||2.3%||2.5%|
|Ill at home during field period||48||0.3%||0.3%|
|Away or in hospital throughout field period||108||0.6%||0.6%|
|Physically or mentally unable/incompetent||153||0.8%||0.9%|
|Other non-response (not covered by categories above)||28||0.2%||0.2%|
|Unable to locate address||26||0.1%|
|Unknown if occupied, due to non-contact||289||1.6%|
|Other unknown eligibility||0||0.0%|
|Total unknown eligibility||320||1.8%|
|Estimated eligible addresses in set of unknown eligibility addresses||304||1.7%||1.8%|
|Total eligible addresses||16,724||92.8%||100.0%|
|Not yet built / under construction||8||0.0%|
|Address occupied but not resident household||214||1.2%|
|Communal establishment / institution||22||0.1%|
|Estimated ineligible addresses in set of unknown eligibility addresses||16||0.1%|
|Total not eligible||1,298||7.2%|
|All issued addresses||18,022||100.0%|
6.3 Drop in response rate
The reduction in fieldwork performance resulted in a response rate of 64% for 2017. This was the same as in previous year and 3 percentage points lower than the 2014 survey response rate of 67%. Response rates are shown in the Table 6.2: Response rates over time below.
Table 6.2: Response rates over time
The long-term average response rate for 1999-2011 was 67.9%. However, it should be noted that the calculation has changed slightly for 2012 as a portion of the addresses of unknown eligibility are considered to be eligible whereas previously they would all have been classed as ineligible. This calculation change would have led to a lower response rate in years prior to 2012, if it had been calculated on the same basis.
Due to a drop in the response rate in 2015, an analysis was conducted to investigate the likely impact of this drop in response rates. This was undertaken by modeling the 2014 sample results to examine what the impact of a 3% drop in response rates would have been had a lower number of interviews been achieved, and comparing this against the 67% full sample results.
Analysis showed that the demographic composition of the sample was largely unchanged (age, gender), with only the most deprived SIMD quintile and other urban areas showing a 1 percentage point drop in their respective shares of the total sample.
Analysis of mean differences in the population estimates from the two samples for a basket of full sample questions from the SHS, and full one third sample questions from the SHCS module, were also undertaken.
This showed that the absolute mean differences for the total population estimates across the different basket of questions within the household and random adults part of the survey, including the Scotland Performs National Indicators, were very small, at around 0.10 percentage points.
A few sub-group categories had one or two 'maximum' differences in estimates of around 1 percentage points, including age and social and private-rented sector sub-groups. However, these differences are unlikely to be statistically significant due to small subgroup sample sizes.
Sub-national analysis was not considered. It is expected that there would be a greater impact of this lower response rate for Local Authorities and other sub-national geographies.
6.4 Local authority performance
Table 6.3: Response rate and eligibility by local authority shows levels of ineligible addresses, response rate and random adult conversion.
Table 6.3: Response rate and eligibility by local authority
As in previous years, the highest levels of ineligible addresses were recorded in Argyll and Bute, Dumfries and Galloway, Na h-Eileanan Siar, Highland and Orkney Islands. For both Na h-Eileanan Siar and Argyll and Bute, high levels of ineligible addresses were expected as both areas contain a high number of holiday and second homes which are not eligible for the survey. Expected levels of ineligible addresses for Dumfries and Galloway, Highland and Orkney were all above 10%.
The two lowest household response rates in 2017 were in Glasgow City and East Renfrewshire, with Glasgow City having the lowest household response rate. In addition, 5 other local authorities had a response rate under 60 per cent in 2016.
The conversion from household interview to random adult completion was 92 per cent in 2017(this was 92% in 2016, 91% in 2015, 92% in 2014 and 93% both in 2012 and 2013).
6.5 Physical survey
As described in section 2.2, over one third of the SHS sample was assigned to the physical survey module. For completion of the physical survey, respondents had to agree to make an appointment for a surveyor to make a follow-up visit and to complete the appointment. Table 6.4 shows the number of households assigned to the physical module which responded to the main SHS and the conversion to completion of the physical survey.
There was a target of at least 80 completed physical surveys for each local authority along with a target of 3,004 surveys for Scotland. The result of continuing lower than estimated levels of conversion from household survey to physical survey, meant that there were fewer than 80 physical survey responses for 9 of the local authorities in 2017. However, the decision to 'open up' an additional stream to the physical survey gave a lower shortfall on the minimum target of 80 per local authority than in previous years (16 local authorities in 2016, 17 in 2015, 24 in 2014, 21 in 2013 and 20 in 2012). The 3,006 surveys achieved for Scotland in 2017 met the target, the first time since prior to 2012.
Table 6.4: Conversion to full physical survey