ANNEX: SCOTTISH ENVIRONMENTAL ATTITUDES AND BEHAVIOURS SURVEY 2008 - TECHNICAL ANNEX ON SURVEY DESIGN
1. Introduction and overview
1.1 This technical annex provides details of the sampling design and its probable impact on the quality and precision of information obtained from the survey. It comments in some detail on the use of a quota sample and its consequences for bias and precision. The quota design appears to have delivered a survey sample of quality comparable to what would have been achieved with a probability sample. The major impact of the design on the precision of estimates from the survey is the clustering of answers to questions within the small areas used as sampling units. Detailed tables for this are provided along with recommendations on how to adjust for the sample design in the analysis.
2. Survey design
2.1 Historically, quota samples have been shown to have the potential to produce very biased results. For example, in a 1945 survey of reading habits 25 where the interviewers selected households to interview and applied quotas by age, sex and socio-economic status, the results obtained were shown to be very different from census returns on key variables, such as education. There are other examples from around that time, including electoral polls, which showed similar results. It is thought that these results were part of the motivation for survey organisations turning to probability sampling. However, more recent developments in survey sampling have modified this view. 26 In theory, the biggest disadvantage of quota sampling is that a biased sample of respondents may be selected. This can happen in two ways:
1. Non-responding individuals or households are ignored and the next one meeting the quota is selected.
2. The interviewer selects the people or households to be interviewed.
These factors can also affect probability samples and non-response is an increasing problem. The common practice of substituting new addresses for non-responders adds to this problem. It can now be argued that a rigorously designed quota sample can be as good as a probability sample for lower cost.
2.2 This survey has used a rigorous approach to quota sampling which should have minimised the extent to which interviewers have a choice in selecting respondents. The first stage of selecting the sampling points was a probability sample stratified by local authority and rurality. For each sampling point, interviewers had a list of addresses within fairly small areas and the quotas were set on four different characteristics, thus limiting the interviewers' choices. Any departures from the planned quotas have been adjusted for in the analysis by reweighting to the marginal totals used to assign them. The survey has also collected a range of other data to assess the quality of the sample (see Chapter 2 of the main report and the following section).
2.3 A criticism of quota samples is that classical sampling theory, based on selection probabilities, cannot be invoked and so no error estimates are available. However, another model based approach can be used to justify the calculation of sampling errors and it leads effectively to the same results as the classical approach 27. The assumption required for this approach is that the sample selected, based on the quota, would be expected to give the same answers to survey questions (on average) as those who would have matched the quotas and were not selected.
3. How well did the sample match the quotas and population totals?
3.1 The quotas specified 8 interviews per sampling point. Exactly 8 interviews were achieved for 70.4% of the sampling points, 9 interviews in a further 10.3% and 7 interviews in a further 5.7%. The quotas varied very little by sex between sampling points, somewhat more by age groups and by full-time or self-employed status, and to the greatest extent by car ownership. Table 1 gives the quotas for the 388 sampling points and the proportions compared to the correct number for points where exactly 8 interviews were obtained.
Table 1 : Quotas and achieved sample for sampling points
Number of interviews in quota for 388 sampling points
Compared to quota for 273 points with 8 interviews
Working full time or self employed
3.2 The small imbalance from the design was corrected via weighting back to the 2001 census household population totals. The most extreme contributions to the weighting were car ownership (car owners average weight 0.93, non car owners 1.16) and ages 25-34 (average weight 1.19) and 55+ (average weight 0.91). Chapter 2 of the main report presents comparisons with the Scottish Household Survey data, a probability sample carried out by the same field force with a very similar design to this survey. The results are very reassuring. Table 2 presents comparisons to the current household population estimates by age and sex 28 and with a more detailed age breakdown than was used in defining the quotas. The standard errors of the percentages from the survey were around 0.5%, so only the over-representation of women aged 25-34 and the under-representation of 55 to 64 year olds was evident. These differences were also seen when comparing with the 2001 census totals.
Table 2: Percentages of population by age and sex sample compared to 2007 household population estimates.
Survey weighted percentages
Population estimated percentages
Difference survey -population
3.3 Although the sampling frame is households, the survey is seeking a population sample. Had it been a probability sample of households it would have been necessary to weight by household size to get population estimates and thus to incur a loss of precision from uneven weights. A stringent test of how well the quotas have delivered a representative sample is to consider household composition for respondents. Table 3 compares the survey and the census by household composition.
Table 3: Percentage of survey (weighted) and of 2001 census populations by number of adults and type of household.
Difference Survey - census
Number of adults
More than one adult
Type of household
Couple no children
Couple with children
The quotas have gone a long way to making the sample representative of people in households, rather than households. Some differences remain with single people being somewhat over-represented and couples without children under-represented. These differences are small and unlikely to have much impact on the results.
4. Impact of the design on the precision of estimates from the survey
4.1 Three aspects of the survey design will influence the precision of estimates: weighting, clustering and stratification. The range of weights here was small (0.81 to 1.44) and this aspect by itself would increase the standard errors of survey estimates by a design factor of only 1.01 (1%). Clustering is the aspect of the survey design that has the largest influence on estimate precision. For estimation of population proportions, means or totals this will differ according to whether the question of interest has answers that are clustered by sampling point. With an average of just under 8 interviews per sampling point, an answer that was maximally clustered would have a design effect of 8 and standard errors increased by a design factor of its square root (2.8). The sampling points for the survey were selected with stratification by local authority and by urban rural classification. This could have the effect of improving the precision of estimates. With 32 local authorities and 6 urban-rural groups, many with no points selected, categories were pooled to provide a set of 50 strata that could be used in the analysis. The stratification had only a very minor influence on the precision of estimates. Adjustment for stratification has been incorporated into the design factors quoted here, but it could have been omitted and very similar results would have been obtained.
4.2 Standard software packages such as SPSS, SAS and STATA now provide options for analysing complex surveys with the appropriate adjustments for survey design, although for SPSS an additional licence is required. We recommend that these programs should be used and details will be provided in the user guide for the survey. For the benefit of people unable to use these programs, the tables in the next section give design factors for estimates of population proportions selected questions. Most questions on environmental attitudes have design effects of 1.3 or lower. Questions on knowledge and awareness had larger design factors, as had some questions on environmental behaviour, generally where these would have been influenced by local circumstances.
4.3 The impact of the survey design on comparisons of proportions or on regression coefficients is generally much lower than their impact on population proportions. The impact depends on both the clustering of the outcome and of the factors that are used in the comparison. If the factor is not clustered the clustering of the outcome will not matter and design factors for regression coefficients will be close to 1.0. When both the outcome and the factor are clustered the standard error of the regression coefficient may be inflated by a design factor, but this will generally be less than those for the individual variables.
4.4 When analyses are carried out on subgroups of the data the design factors are generally reduced compared to what they would have been for the full data set. This comes about because the effective cluster size is smaller than that for the full survey. Tables 6.2 and 6.3 of the Survey Technical Report are examples of regression analyses for sub-samples. We can see that the survey design has had very little effect on the precision of estimates compared to what would have been achieved from a simple random sample. The only factor where it makes any real difference to the standard error of the coefficient is housing type, which is likely to be highly clustered by sampling point and the urban rural indicator in Table 6.2.
4.5 Design factors for demographic variables are given below to guide an analyst without the ability to use an appropriate survey analysis package. These are only general rules and individual analyses may differ. For example, to test a worst-case scenario, a comparison between a highly clustered item (awareness of Scotch beef) was compared by a very clustered factor (urban/rural classification). Most regression coefficients had design factors close to 1.0, when a larger one was expected. This was because Scotch beef awareness was only clustered in urban areas with virtually 100% of rural respondents reporting being aware of it.
5. Tables of design factors
Design factors for all-Scotland estimates of environmental attitudes, knowledge and behaviours
5.1 These tables give the design factors for the proportions in the population of selected measures of environmental knowledge, attitudes and behaviour as well as some important demographic variables. The selection of design factors to present was based on those highlighted in the main report. Design effects are not quoted for categories affecting only a very small proportion of respondents. Such measures would almost always have design factors very close to 1.0 unless they were very localised (e.g. people who read the Dundee Courier) and thus affected by clustering. To obtain standard errors and confidence intervals for the population proportions, the procedure is as follows:
- Calculate the weighted proportion (p) or percentage (P).
- Get the standard error of p or P from the usual formulae or where N is the sample size (here usually 3054).
- Multiply the standard error by the design factor and calculate confidence intervals from the new value as usual.
Figures 3.1 and 3.2 in the main report (Questions B1 and B2) present data on how importantly people rated various environmental issues.
Table 4: Design factors for selected entries in Figures 3.1 and 3.2
How important was each of these issues
Facing the world
Most important issue
Most important issue
Economy / credit crunch
Crime / anti-social behaviour
Unemployment / lack of industry
Environment/ climate change/ global warming
Environmental factors have a fairly low design factor and this was confirmed by the design factor for the salience scale where the category with the highest design factor (1.2) was for those with the lowest salience (no mention).
5.2 None of the environmental attitudes reported in chapters 4 or 5 of the main report have large design factors, with the majority being around 1.1. The classification of 'greenness' is given as an example in Table 5.
Table 5: Design factors for the hierarchical classification
|% in group||Standard Error|
There were substantial design factors for the extremes of environmental knowledge (1.40 for "Heard of climate change but know nothing about it" and 1.92 for "Never heard of climate change"). But questions about detailed knowledge for those who knew something about climate change all had design factors around 1.0. Design factors were more pronounced for awareness of specific items and for some environmental behaviours. Design factors for the environmental behaviours from Tables 5.9 (Q23) and 5.18 and 5.19 (Q 25) of the main report are shown in Table 6. Some behaviours have large design factors, probably due to the clustering of housing types or of local provision of services.
Table 6: Design factors for behaviours from Questions 23 and 25.
Design factor for answers to question
How often do you personally do the following
Cannot / service not provided / not applicable
Q23 Table 5.9
Turn off heating
Turn off tap
Use energy saving bulbs
Hang out washing
Turn off lights
Avoid filling kettle
Q25 (Tables 5.18 and 5.19)
Composting (those with gardens)
Kerbside garden waste
Kerbside bottle recycling
Other bottle recycling
Kerbside can recycling
Kerbside paper recycling
Other paper recycling
Q26 Table 5.23
Reuse plastic bottles
Plastic food containers
Reuse wrapping paper
Own shopping bags/boxes
Donate to charity shops
In chapter 6 of the main report questions on sources of information have design factors in the range 1.2 to 1.4, while the corresponding questions on degree of trust have lower factors effects between 1.0 and 1.3. Questions on alternative energy sources (Figure 6.3, Q E3) also had design factors in the range 1.0 to 1.3, except for the "don't know" option with higher values at around 1.4. Questions on support for government policies (Figure 6.4 QC42) all have design factors of around 1.2.
Design factors for demographic variables
5.3 Design factors for demographics are not important in themselves, but they determine how comparisons and regressions with demographic variables will be affected by the sample design. The balance by sex gave the proportions of male or female a design factor of 0.68; while those for age groups, number of adults, number of children and proportion working full-time were all close to 1.0. This means that no design adjustment is needed for comparisons or regression coefficients using these variables, indeed comparisons by sex will be more precise than the equivalent random sample.
The demographic variable most affected by clustering was the urban/rural indicator, since it is defined at the cluster level. Due to clustering alone this has a design factor of 2.8 (the square root of the average cluster size). The design factors for other demographic variables are given in Table 7.
Table 7: Design factors for demographic variables with appreciable design factors.
A B C1 vs others
own outright or mortgage
Local Authority rent
Rent housing association etc
A house or bungalow
A flat, maisonette or apartment
6. Conclusion to Technical Annex
6.1 The quota design for the SEABS'08 appears to have delivered a survey sample of quality comparable to what would have been achieved with a probability sample. The major impact of the design on the precision of estimates from the survey is the clustering of answers to questions within the small areas used as sampling units. Detailed tables for this have been provided in this annex, along with recommendations on how to adjust for the sample design in the analysis.