Publication - Research and analysis

Scottish Household Survey: response rates, reissuing and survey quality

This paper assesses the impact of reissuing on survey estimates using data from the Scottish Household Survey, 2014 and 2016.

55 page PDF

1.5 MB

55 page PDF

1.5 MB

Contents
Scottish Household Survey: response rates, reissuing and survey quality
4 Results

55 page PDF

1.5 MB

4 Results

4.1 We briefly summarise the difference in the profile of people who respond at first issue to those who respond at reissue at the start of this section before examining the impact of excluding reissues on survey estimates.

How do those that respond at first issue differ from those who respond at reissue?

4.2 Reissues accounted for 16% of random adult interviews in 2014 and 18% in 2016.

Table 4.1: Profile of first issue random adult respondents compared with reissue respondents. SHS 2014 and 2016 unweighted
SHS
2014 2016
First issue Reissue Final N First issue Reissue Final N
Male 45% 45% 45% 4,442 45% 48% 46% 4,401
Female 55% 55% 55% 5,356 55% 52% 54% 5,241
Total 100% 100% 100% 9,798 100% 100% 100% 9,642
16 - 24 7% 11% 8% 787 8% 8% 8% 727
25 - 44 28% 33% 28% 2,787 27% 36% 29% 2,752
45 - 59 26% 26% 26% 2,532 25% 25% 25% 2,379
60+ 39% 29% 38% 3,692 41% 32% 39% 3,784
Total 100% 100% 100% 9,798 100% 100% 100% 9,642
Urban 78% 88% 79% 7,752 77% 85% 78% 7,528
Rural 22% 12% 21% 2,046 23% 15% 22% 2,114
Total 100% 100% 100% 9,798 100% 100% 100% 9,642
15% most 14% 20% 15% 1,476 13% 17% 14% 1,349
Rest 86% 80% 85% 8,322 87% 83% 86% 8,293
Total 100% 100% 100% 9,798 100% 100% 100% 9,642
Single adult 18% 26% 19% 1,850 17% 26% 19% 1,807
Small adult 15% 16% 15% 1,494 15% 15% 15% 1,446
Single parent 5% 8% 5% 526 6% 8% 6% 570
Small family 12% 11% 12% 1,171 12% 11% 12% 1,153
Large family 6% 4% 6% 549 5% 4% 5% 433
Large adult 8% 7% 8% 792 8% 6% 7% 707
Older small 18% 12% 17% 1,659 19% 14% 18% 1,725
Single pensioner 18% 15% 18% 1,757 19% 16% 19% 1,801
Total 100% 100% 100% 9,798 100% 100% 100% 9,642
N 8,273 1,525 9,798 7,923 1,719 9,642

4.3 The profile of people who respond at reissue is broadly in line with the profile of those who respond at first issue in both years and the differences are not stark. Table 4.1 show the unweighted distributions across key sub-groups by when interviewed.

4.4 In both waves, men and those who were in the younger age bands comprised a higher proportion of reissue interviews than first issue interviews. This echoes findings from previous research reported in Chapter 2.

4.5 In relation to rurality and deprivation, reissue interviews were more likely to occur in urban areas, and in the 15% most deprived areas than first issue interviews were.

4.6 With regard to household type, Single Adult and Single Parent households comprised a higher proportion of reissue interviews than first issues, while the opposite was the case for Older Smaller households and Single Pensioner households.

What is the impact of reissuing on national estimates?

4.7 Table 4.2 shows the impact of reissuing on twelve estimates at the Scotland-wide level for the 2016 wave of the SHS.

4.8 The difference between the final sample estimates and the issue 1 survey estimate was small in absolute terms.

4.9 The average absolute difference between the final sample and the first issue estimates was 0.33 percentage points, and the maximum difference was 1.13 percentage points.

4.10 Table 4.2 also shows the standardised differences, the ratio of the absolute difference between estimates to the standard error of the main estimate. Overall, the average value of this ratio was 0.58 across all 12 estimates. In other words, the average impact of not including reissues equates to just over half of one standard error of the published estimates. The maximum value of this ratio among the twelve measures was 2.07, for the measure of providing unpaid help to organisations or groups within last 12 months.

Table 4.2: Impact of reissues on twelve key national estimates. SHS 2016
Final estimate (64% RR) N[6] SE CIs (+/-) Issue 1 estimate (54% RR) Difference Diff/SE
Percentage satisfied (very or fairly) with local public services 56.1% 9,594 0.6% 1.2% 55.9% 0.23% 0.38
Percentage agreeing (strongly or slightly) they 'can influence decisions affecting my local area' 23.1% 9,642 0.5% 1.0% 23.3% 0.15% 0.30
Percentage using the internet for personal use 83.4% 4,707 0.7% 1.3% 84.1% 0.74% 1.13
Percentage rating neighbourhood as a very good place to live 56.7% 9,642 0.6% 1.2% 56.6% 0.05% 0.08
Percentage participating in a cultural activity or attended a cultural place or event in the last 12 months 92.0% 5,008 0.5% 0.9% 92.0% 0.01% 0.03
Percentage that make 1+ visits to the outdoors per week 48.5% 9,642 0.6% 1.2% 48.1% 0.35% 0.57
Percentage living within 5 min walk of greenspace 65.4% 9,642 0.6% 1.1% 66.4% 1.07% 1.84
Percentage providing unpaid help to organisations or groups within last 12 months 27.3% 9,642 0.5% 1.1% 28.4% 1.13% 2.07
Percentage participating in physical activity or sport in last four weeks 78.8% 9,642 0.5% 1.0% 78.8% 0.01% 0.02
Percentage rating general health as bad or very bad 7.7% 9,642 0.3% 0.6% 7.8% 0.07% 0.21
Percentage experiencing either discrimination or harassment 9.8% 9,642 0.4% 0.7% 9.9% 0.07% 0.20
Percentage households not managing well financially 8.2% 10,470 0.3% 0.6% 8.1% 0.04% 0.13
Average across the twelve measures. 0.33% 0.58

4.11 The results of the same analysis conducted on the 2012-2013 wave of the data presents a similar picture (Table 4.3). In terms of the absolute differences, the average among the 12 measures was 0.44 percentage points and the maximum difference was 0.93 percentage points.

4.12 With regard to the standardised measure, the average value of the ratio of the difference to the standard error of the main estimate was 0.86 across the 12 estimates, with a maximum of 1.66 for the measure, 'Percentage living within 5 min walk of greenspace'. Again, this means that the impact of not including reissues in the survey (and reducing the effective response rate by around 10%) would equate to less than one standard error for most measures.

Table 4.3: Impact of reissues on twelve key national estimates. SHS 2014
Final estimate (67% RR) N[7] SE CIs (+/-) Issue 1 estimate (56% RR) Difference Diff/SE
Percentage satisfied (very or fairly) with local public services 61.9% 9,746 0.6% 1.2% 61.2% 0.74% 1.26
Percentage agreeing (strongly or slightly) they 'can influence decisions affecting my local area' 23.0% 9,798 0.5% 1.0% 23.1% 0.16% 0.31
Percentage using the internet for personal use 82.0% 4,787 0.7% 1.3% 82.5% 0.55% 0.83
Percentage rating neighbourhood as a very good place to live 55.8% 9,798 0.6% 1.2% 55.5% 0.38% 0.63
Percentage participating in cultural activity/ attending a cultural place/event in the last 12 months 90.8% 5,140 0.5% 0.9% 91.0% 0.15% 0.32
Percentage that make 1+ visits to the outdoors per week 48.4% 9,798 0.6% 1.2% 47.9% 0.48% 0.79
Percentage living within 5 min walk of greenspace 68.6% 9,798 0.6% 1.1% 69.5% 0.93% 1.66
Percentage providing unpaid help to organisations or groups within last 12 months 27.0% 9,798 0.5% 1.1% 27.5% 0.52% 0.96
Percentage participating in physical activity or sport in last four weeks 77.8% 9,798 0.5% 1.0% 78.2% 0.44% 0.88
Percentage rating general health as bad or very bad 6.6% 9,798 0.3% 0.6% 6.4% 0.15% 0.51
Percentage experiencing either discrimination or harassment 8.9% 9,798 0.3% 0.7% 9.2% 0.31% 0.91
Percentage households not managing well financially 11.2% 10,632 0.4% 0.7% 10.7% 0.47% 1.29
Average across the twelve measures. 0.44% 0.86

4.13 The scale of the impact of not including reissues, our proxy for examining the effect of setting a lower response rate target, is easier to visualise as estimates plotted with confidence intervals. Figures 4.1 to 4.4 show estimates of four of the measures plotted with confidence intervals.[8]

  • Satisfaction with local public services
  • Providing unpaid help to organisations or groups within last 12 months
  • Percentage participating in physical activity or sport in last four weeks
  • Percentage households not managing well financially.
Figure 4.1: Percentage very or fairly satisfied with local services by year and by whether reissues included in data. SHS 2014 and SHS 2016 [9]
Figure 4.1: Percentage very or fairly satisfied with local services by year and by whether reissues included in data. SHS 2014 and SHS 2016
Figure 4.2: Percentage providing unpaid help to organisations or groups within last 12 months by year and by whether reissues included in data. SHS 2014 and SHS 2016
Figure 4.2: Percentage providing unpaid help to organisations or groups within last 12 months by year and by whether reissues included in data. SHS 2014 and SHS 2016
Figure 4.3: Percentage participating in physical activity or sport in last four weeks by year and by whether reissues included in data. SHS 2014 and SHS 2016
Figure 4.3: Percentage participating in physical activity or sport in last four weeks by year and by whether reissues included in data. SHS 2014 and SHS 2016
Figure 4.4: Percentage households not managing well financially by year and by whether reissues included in data. SHS 2014 and SHS 2016
Figure 4.4: Percentage households not managing well financially by year and by whether reissues included in data. SHS 2014 and SHS 2016

4.14 Table 4.4 presents the summary of the absolute impact of reissuing on the 12 national estimates. The average impact was 0.38 percentage points. This was similar between 2014 (0.44 percentage points) and 2016 (0.33 percentage points). Most of the estimates, 21 of 24, changed by less than 1 percentage point.

Table 4.4: Summary of absolute impact on the national estimates. SHS 2014 and SHS 2016
Mean Max Count Final minus Issue 1 estimate 0-1% points Final minus Issue 1 estimate >1% points Final minus Issue 1 estimate >3% points
2014 0.44% 0.93% 12 12 0 0
2016 0.33% 1.13% 12 9 3 0
Combined 0.38% 1.13% 24 22 3 0

Sample sizes: See Table 3.1.

4.15 Table 4.5 shows a summary of the impact on these estimates after standardisation. Overall, the average standardised impact on estimates was 0.72. This means that the impact of reducing the response rate by around 10% to 11% is of a similar magnitude to around three-quarters of the standard error associated with the estimates.

Table 4.5: Summary of average standardised impact of reissuing on national estimates. SHS 2014 and SHS 2016
Mean Max Count Diff/SE 0 to 0.5 Diff/SE >0.5 Diff/SE >1 Diff/SE >1.5
2014 0.86 1.66 12 2 10 3 1
2016 0.58 2.07 12 8 4 3 2
Combined 0.72 2.07 24 10 14 6 3

Sample sizes: See Table 3.1.

Analysis of impact of reissuing on estimates among key subgroups.

4.16 While the impact of reissuing on estimates at the national level was small, the impact on estimates among sub-groups could potentially be more considerable. Estimates for the 11 measures from the random adult section of the questionnaire were analysed. The impact on these estimates was calculated on a number of key sub-groups. These were gender, age, rurality, deprivation, tenure, area and household typology.

4.17 Overall, this meant that the impact was calculated for 704 estimates, 352 in 2014 and 352 estimates in 2016.[10] Table 4.6 summarises the impact on the absolute difference of estimates among key subgroups.

Table 4.6: Summary of absolute impact on estimates among key subgroups. SHS 2014 and 2016
Mean Max Count Final minus Issue 1 estimate 0-1% Final minus Issue 1 estimate >1% Final minus Issue 1 estimate >3% Final minus Issue 1 estimate >5%
2014 0.70% 5.5% 352 277 75 4 1
2016 0.82% 12.7% 352 249 103 8 1
Combined 0.76% 12.7% 704 526 178 12 2

4.18 Overall, reissuing had a larger impact at the sub-group level than the national level. This is primarily driven by the sample sizes. However most of the differences were still relatively modest. Overall, the average impact on estimates was 0.76%. The average impact was similar in 2014 (0.70%) and 2016 (0.82%).

4.19 Most estimates, 526 of 704, changed by less than 1%. Overall only 12 of the 704 estimates changed by more than 3%, and two changed by more than 5%. Estimates for Central Region[11] accounted for 6 of the 12 estimates that changed by more than 3 percentage points. Similarly, 6 of the 12 estimates that changed by more than 3 percentage points related to the indicator for making one or more visits to the outdoors per week.

4.20 By way of illustration, Figure 4.5 shows the estimates for making one or more visits to the outdoors per week for Central Region. In 2016 this estimate changed by 4% for Central Region, from 38.4% to 42.4%. The overall sample size for the main estimate was based on 754 cases. This means that, as shown in Figure 4.5, the confidence intervals around this estimate equalled +/- 4.2%, and therefore that the standardised change equated to 1.88 of the standard error of the main estimate.

Figure 4.5: Percentage saying that they make one or visits to the outdoors per week in Central Region. SHS 2014 and 2016
Figure 4.5: Percentage saying that they make one or visits to the outdoors per week in Central Region. SHS 2014 and 2016

4.21 Finally, Table 4.5 shows a summary of the impact on these 704 estimates after standardisation.

Table 4.5: Summary of average standardised impact of reissuing on estimates among key subgroups. SHS 2014 and 2016
Mean Max Count Diff/SE 0 to 0.5 Diff/SE >0.5 Diff/SE >1 Diff/SE >1.5
2014 0.48 2.76 352 218 134 34 3
2016 0.53 6.21 352 215 137 49 17
Combined 0.51 6.21 704 433 271 83 20

4.22 Overall, the average standardised impact on estimates was 0.51. This means that the impact of reducing the response rate by around 10% is of a similar magnitude to one half of the standard error associated with the estimates. There is little difference with regard to the size of the impact by wave.

4.23 The impact was less than 0.5 for the majority of estimates (433 of 704).

4.24 The impact was greater than 1.5 for less than 3% of estimates (20 out of 704 estimates. The maximum value was 6.21 for all sub-group estimates included in this analysis and the second largest value was 2.76.

4.25 There was some evidence to suggest that the impact of reissuing on some sub-groups was greater than on others. However, this appears to be driven, at least in part, by the proportion of reissue interviews within particular subgroups. The maximum of the average standardised impact across the different measures but within sub-groups was 0.94 in the 2014 wave and 1.82 for 2016. These both related to estimates for the Central Region of Scotland. Note that 21% of interviews in Central Region were reissue interviews in 2014 while 27% were reissues in 2016[12] compared to the overall average of 16% in 2014 and 18% in 2016.

4.26 The second largest average standardised impact across both years was for single adult households (0.74 in 2014 and 1.03 in 2016). Reissue interviews also accounted for a higher proportion of interviews among this sub-group than most other sub-groups (22% in 2014 and 25% in 2016).

4.27 As noted previously, the scale of the impact can be difficult to visualise. Figures 4.6 to 4.10 show estimates for five sets of sub-group estimates plotted with confidence intervals

  • Satisfaction with public services among those in the most deprived 15% of areas
  • Rate neighbourhood as a good place to live in Fife
  • Rating general health as bad or very bad among those aged 60 and over
  • Participated in a physical activity or a sport among those aged 16-24
  • Providing unpaid help to organisations or groups within last 12 months among men
Figure 4.6: Percentage of those living in the most deprived 15% of areas who are satisfied with local public services by year and by whether reissues included in data. 2014 and 2016
Figure 4.6: Percentage of those living in the most deprived 15% of areas who are satisfied with local public services by year and by whether reissues included in data. 2014 and 2016
Figure 4.7: Percentage of those in Fife who rate their neighbourhood as a good place to live by year and by whether reissues included in data. 2014 and 2016
Figure 4.7: Percentage of those in Fife who rate their neighbourhood as a good place to live by year and by whether reissues included in data. 2014 and 2016
Figure 4.8: Percentage of those aged 60 and over who rate their general health as bad or very bad by year and whether reissues included in data. 2014 and 2016
Figure 4.8: Percentage of those aged 60 and over who rate their general health as bad or very bad by year and whether reissues included in data. 2014 and 2016
Figure 4.9: Percentage of those aged 16-24 who have participated in a physical activity or a sport by year and whether reissues included in data. 2014 and 2016
Figure 4.9: Percentage of those aged 16-24 who have participated in a physical activity or a sport by year and whether reissues included in data. 2014 and 2016
Figure 4.10: Percentage of men who have provided unpaid help to organisations or groups with the last twelve months by year and whether reissues included in data. 2014 and 2016
Figure 4.10: Percentage of men who have provided unpaid help to organisations or groups with the last twelve months by year and whether reissues included in data. 2014 and 2016

Contact

Email: shs@gov.scot