3 QUESTIONNAIRE DESIGN
Questionnaire content and structure
3.1 The starting point in terms of identifying content for the SEABS'08 survey questionnaire was DEFRA's (2007) ' Survey of Attitudes, Knowledge and Behaviour in relation to the Environment'10 and a related DEFRA-commissioned omnibus survey module. Following a review of these questionnaires and other relevant studies, together with the SEABS'08 study objectives, Ipsos MORI Scotland, working in collaboration with the SG Project Management Team and the Greener Scotland and Climate Change Divisions, produced a draft SEABS'08 questionnaire. The draft was presented to the SEABS'08 Scottish Government Advisory Group, and the Scottish Environment Social Evidence Group 11 ( SESEG) for discussion. It was subsequently refined and a revised version agreed.
3.2 Specific topics covered in the questionnaire were as follows:
- the salience of the environment and of specific environmental issues;
- awareness of, and attitudes towards, climate change;
- travel behaviour;
- energy consumption;
- reusing and recycling;
- wellbeing; and
- use and perceived importance of greenspace.
3.3 In addition to the data requirements set out in the project brief, four key considerations underpinned decisions on the choice of survey content. Firstly, to ensure the efficient use of space in the survey, it was important to avoid simply duplicating questions from other quantitative research studies that have been conducted over recent years - for example, the Scottish Waste Awareness Group has undertaken various surveys looking in detail at the public's recycling behaviour. Secondly, all questions in the survey had to be policy-relevant; that is, they needed to focus on issues that are of particular interest for the Scottish Government. Thirdly, and where relevant, the survey questions had to be tailored to the Scottish context - this was particularly relevant in relation to items focusing on energy use in the home and transport, where there was reason to believe that the behaviour of Scots might differ markedly to that of people living elsewhere in the UK. Fourth, because there is an intention that the survey will be repeated in the long term, it was important that the questionnaire was 'future proof' and did not include questions that are likely to be irrelevant in a few years time.
3.4 In designing the questionnaire, it was important to be mindful of the potential effects of social desirability bias. Social desirability effects refer to evidence that survey respondents' answers to questions are, at times, related to their perceptions of the social acceptability or political correctness of their answers. Therefore, the survey was not introduced to respondents as being about the environment specifically but more generally about "what the people of Scotland do and think". This was to try to ensure that those who were more enthusiastic in their environmental viewpoint were no more or less likely to participate than those with a more sceptical or disengaged attitude towards the environment.
3.5 The structure of the questionnaire was designed to elicit as accurate as possible an account of public engagement with the environment issue. The core, environmental part of the questionnaire was divided into three consecutive sections. The first section focused on participation in environmental behaviours; the second on barriers to participation in environmental behaviours; and the third on environmental attitudes. There were two key reasons for this choice of ordering. First, the barriers questions were designed to be asked of respondents who reported that they did not or rarely performed particular environmental behaviours. Had these questions been asked in tandem with the questions measuring participation in environmental behaviours, rather than being kept entirely separate, respondents may have grown used to this pattern as the interview progressed and realised that they could avoid having to answer barriers questions by giving 'green' responses to the behavioural questions. Second, the attitudinal questions were placed after the behavioural questions so that respondents would not be tempted to 'adjust' their self reported levels of participation in environmental behaviours to bring these into line with any green attitudes they expressed earlier in the questionnaire.
3.6 To minimise such social desirability effects, a section of the survey covering some of the attitudinal questions in the questionnaire was designed to be administered via CASI (Computer Assisted Self-Interviewing), whereby the respondent is invited to key their answers directly into the CAPI machine.
3.7 Cognitive testing of the questionnaire was undertaken to ensure that all questions and response options were understood in the way intended. Cognitive question testing is a helpful process in assessing whether a survey instrument will be successful in eliciting the required information. In essence, it tests whether respondents will understand the questions and response options in the way that they are intended.
3.8 The cognitive testing was undertaken among 20 members of the public between 19th May and 6th June 2008. The sample was drawn from the Scottish Household Survey ( SHS) respondent database which is comprised of approximately 10,000 individuals who have taken part in the SHS within the last two years and agreed to be re-contacted for future research exercises. The sample was selected to include a cross-section of the public in terms of age, sex, household type and size, car ownership and location.
3.9 Members of the sample were telephoned and asked if they would be willing to participate in the study; and a convenient time was arranged for an interview. The 20 respondents who took part included:
- ten Men and ten women;
- seven people aged 16 to 34 years, seven aged 35 to 59 years and six aged 60 years and over;
- ten people living in an urban area (seven in Edinburgh and three in Glasgow), five in semi-urban areas and five living in a rural location;
- thirteen people living in a house and seven living in a flat; and
- twelve people with access to a car and eight without.
3.10 All interviews were carried out face to face by a core member of the Ipsos MORI project team. A combination of interviewer observation and retrospective probing techniques were used to identify potential problems with the questionnaire. For the testing of attitudinal questions, 'think aloud' techniques were also used.
3.11 On completion of the cognitive testing, Ipsos MORI submitted a report to the Scottish Government which summarised the main findings to emerge from the exercise, together with recommendations for changes to the questionnaire. Most of these changes were minimal and typically took the form of slight alterations to the wording of questions and precodes. All of the recommendations were accepted by the Scottish Government.
3.12 The revised questionnaire was piloted between 21 st July and 3 rd August 2008 to provide a final check of the clarity of questions and to ensure that the CAPI script had been correctly programmed.
3.13 A total of 13 pilot sample points were drawn across Scotland, with the aim of achieving 100 interviews in total (eight in each point). The sample was not intended to be representative of the Scottish population but rather was drawn purposively to ensure the questionnaire would be tested among a range of respondents in different areas of Scotland (urban/rural, north/south). However, reflecting the approach that would be used in the main survey, quotas were set on age, sex, working status and car ownership.
3.14 The pilot interviews were undertaken by a selected team of experienced Ipsos MORI survey interviewers. In advance of fieldwork, the interviewers were briefed by telephone by a member of the Ipsos MORI project team. 103 pilot interviews were achieved.
3.15 On completion of the fieldwork, the interviewers were invited to a debriefing session where they were asked to provide feedback on the questionnaire. On the basis of their comments, Ipsos MORI produced a pilot report for the Scottish Government which made recommendations for some final amendments to the questionnaire. Again, these amendments were minimal and all were agreed by the Scottish Government.