Chapter Three The scenario-based approach
3.1 In this chapter we describe the methods used to develop and test the tools and materials with young people. First we outline scenarios as a research and development method, and then describe how literature and practice were drawn on to create the scenarios and the prototype tools. Then we describe the scenarios and materials used with the focus groups, and key features of the tools themselves.
Scenarios as a research and development method
3.2 Scenario-based materials were used to test the suitability of the tools in workshops with young people, involving a process that was more structured than focus groups typically are 6 . Scenario-based workshops are used in ICT development and socio-technical research, as a means of identifying system requirements and evaluating progress in meeting them, by representing design ideas in an easy to understand non-technical format (Carroll and Rosson 1992, Rosson and Carroll 2001). In this study, scenarios were used to describe in a short narrative form how various ICT-based prototype tools could be used in two contexts; organised groups with face-to-face facilitation ( e.g. at school, or a community centre youth group), or as individuals acting on their own accord at home or elsewhere.
3.3 We briefly outline the basis of the approach here before detailing the factors we took into account and the concrete steps to apply the approach. We intend to show how our approach fits with qualitative social research methods that readers may be more familiar with. The scenario-based approach has been used by the authors previously in e-democracy contexts. The approach complements, on the one hand, action research to develop and improve prototypes with user feedback and, on the other, field study methods to analyse how they are used in day-to-day practice (Whyte and Macintosh, 2003a).
3.4 Research methods for identifying peoples' needs for ICT-based tools and evaluating these have since the late 1980's increasingly overlapped with qualitative social research methods ( e.g. Bannon, 1991). A growing range of studies, especially since (Suchman, 1987) have been influenced by action research and ethnographic field study. Elements of these approaches have been incorporated into scenario-based ICT research and design methods (Rosson and Carroll, op. cit.) and we highlight the following similarities: -
- Stakeholders and intended end-users are identified through negotiation with the project 'owners', and a shared understanding of high-level goals is agreed.
- The research aims to identify feasible and desirable courses of action in collaboration with participants drawn from those groups, through scenario workshops - a structured form of focus group discussion.
- Key themes of the study are developed from an initial set of categories, by testing ideas and refining them from analysis of the responses from participants.
- Analysis also involves critical reflection during and after the researchers' actions to change a situation.
3.5 The major difference between ICT-based applied research and other forms of social research is (naturally) the focus on a 'tool', some software running on a computer or other device, and its relation to the human activities it is meant to support. Field studies of these activities involve observations and analysis of the day-to-day practices and problems that the intended users encounter, before and after new tools are introduced.
3.6 In many cases, as in the present study, semi-structured interviews are a useful source of past experiences and views on what needs improved, but there is little time or opportunity for detailed field observations. Scenarios can be particularly valuable in such cases. If they successfully convey how the designers envisage their prototype tools being used in practice, this better enables the would-be users to identify the likely pros and cons of particular features. Then the analysis of scenario responses can identify the trade-offs that need to be made to make appropriate design decisions about those features (Rosson and Carroll, 2001).
Developing the Scenarios and prototypes
3.7 The essential elements of the approach involve creating:-
- Characters: representing the anticipated 'actors', in terms of the demographic and biographical characteristics that define them as a stakeholder.
- Places/ events: representing the circumstances in which we envisage the tools being used in the engagement process.
3.8 A scenario is simply a sequence of events where the characters use the tools in a context, expressed in written form but preferably also with illustrations (as in a storyboard) or working prototypes that illustrate what the scenarios refer to.
3.9 The scenarios aim for participant feedback on pros and cons of tools that have been designed to a prototype stage, and which make assumptions about the trade-offs that will be needed. ICT tool design requires consideration of, and trade-offs between, many technical and social factors that may affect how the tools can be used to accomplish the desired aims, and whether their intended users will actually use them.
3.10 It is important then for us to detail the considerations that went into the scenarios and the design decisions that shaped the tool prototypes. In the remainder of this section we consider how the 'key dimensions' of the cases reviewed in Chapter 2 were drawn on in order to base the scenarios on analysis of good practice elsewhere 7 .
3.11 Note that some dimensions apply to the current study as well as to the proposed e-engagement process that the study is about. However unless stated otherwise we are referring to that e-engagement process.
Actors: who should be engaged and by whom
3.12 To better consider the organisational aspects of the e-engagement process it should help to define the range of actors it may involve. Table 3.1 below gives a non-exhaustive list of actors and the roles we expect them to take in both the current study and the engagement process itself. Note that individuals and groups can fall under several categories depending on their role. Definitions marked * in the table are from 'Securing Public Confidence in Radioactive Waste Management, Participatory Methods Working Paper' (Chilvers et al, 2003).
Level of engagement
3.13 The 'level of engagement' refers to the level of policy detail, and the weight given to the participants' responses.
3.14 In the e-engagement process we have assumed that the level of engagement will vary through the duration of CoRWM's work and, since there may be parallel needs for the Scottish Executive to consult on other RWM areas, such as the establishment of the Nuclear Decommissioning Authority, several levels of engagement may need to be managed at the same time.
3.15 The design of the tools should not pre-determine the degree of 'weight' given to the participants' input. For example a polling tool might be used referendum-style so that the results determine an outcome, or at the other end of the scale merely as a snapshot of opinion that has no direct bearing on outcomes.
3.16 The youth participation literature indicates that the relationship between results and outcome ought to be made explicit at the outset of the engagement process (even if the results will not directly affect the outcome it is better to say so than pretend otherwise). This can be addressed in the introductory text on a website. It should also be straightforward for consultation 'owners' to update that text without extensive technical skills.
Define overall purpose and control its resources.
Scottish Executive Ministers, CoRWM
Scottish Executive Ministers
Controls the process, sets boundaries.
Same + Youth reference group
SEERAD, Steering Group
Immediate users of the results.
Organisations or individuals contracted to contribute to the engagement process.
Research & consultancy organisations, ICT suppliers, content contributors from stakeholder groups.
Intermediaries between suppliers and stakeholders.
Community & youth sector organisations; COSLA, Local authorities; SIPS; Schools.
Scottish Youth Parliament, Highland Youth Voice; Young Scot/ Dialogue Youth.
Intended/actual participants who contribute directly to the results. Sub-divided as below.
* Scientists, technologists, social scientists, procedural experts
Steering Group, CoRWM + others to be defined
Steering Group, Phase 1 Interviewees.
Departments of the UK Government, Scottish Executive and other devolved administrations, Local authorities, regulatory and other non-departmental bodies.
SEERAD, Civic Participation & Consultation Research Unit. Other depts and agencies, Community Planning groups & others t.b.d.
SEERAD, Civic Participation and Consultation Research Unit
* Public participants in detailed appraisal process, selected to maximise broader societal representation
Users in facilitated groups organised through stakeholders.
Members of recruited focus groups.
* Other individuals not otherwise engaged above.
Self-directed individuals accessing engagement tools on the Internet.
Visitors to Glasgow Science Centre exhibition
Commercial and other organisations involved in RWM.
Nirex, NRPB + others t.b.d.
Nirex, NRPB (background info)
* Individuals or groups representing those who live/work in specific locales including near existing nuclear sites.
School pupils, local community groups in relevant locations.
N. Lanarkshire, Highland, East Lothian area schools & groups
* Professional stakeholders who possess relevant specialist knowledge & represent different economic, political, environmental and social interests.
Scottish Youth Parliament, Young Scot/ Dialogue Youth + others t.b.d.
Scottish Youth Parliament, Young Scot/ Dialogue Youth
Stage in decision-making
3.17 The 'stage in decision-making' refers to when the participants are/were engaged in relation to the policy cycle. The two main aspects that need to be conveyed to young people in the engagement process are: -
- Where does the young people's engagement process fit in relation to the PSE process as a whole?
- Where does the PSE process fit in the overall decision-making?
3.18 The final toolkit should include a clearly defined map showing the schedule of events in the form of a graphical time-line or calendar.
Figure 3.1 Policy life-cycle (from Macintosh, 2004a)
3.19 In terms of our 'policy life-cycle' stages we can assume that all stages will need to be addressed and, at any particular point in time, there may be different consultations at different stages, i.e.
1. Agenda setting: there may be a need to consult on issues that for the e-engagement purposes are still at an 'open-ended' stage, in that the consultation owners do not want the dialogue to be shaped by specific policy statements.
2. Analysis: where the consultation owner has defined an agenda, can state a position relating to the items on that agenda, and is seeking evidence and knowledge from others.
3. Creating the policy: where there is a draft document for formal consultation, potentially greater need to target and identify specific stakeholders/participants, and possibly with options to be assessed in more detail e.g. through simulation of decision-making, or risk analysis.
4. Implementing the policy: likely to include more specialised policy documents with a greater onus on ensuring that respondents have read and understood them; plus a greater variety of supporting information with a more complex relation to original issues, plus other issues that are engendered.
5. Monitoring the policy: this can involve evaluation and review of the policy in action, research evidence and views of users. This may involve going back to stage 1, for further 'agenda-setting' dialogue where most issues are likely to directly relate to those discussed throughout the previous stages.
3.20 The need to consult at successive stages of policy-making suggests that those planning the engagement process need to consider re-using information ( i.e. the engagement materials) at different stages. Furthermore:-
- To manage the information effectively it should be structured around those elements that are least likely to change. In a controversial area such as RWM those elements seems likely to be the main issues or questions, rather than information representing facts or positions on them.
- As the information evolves through dialogue with particular stakeholder groups over a given period, the consultation organiser will need easy-to-use facilities to archive, edit, and update it. Otherwise any requirement for updates to be carried out by an external supplier could act as a delaying factor.
Rules of engagement
3.21 The 'rules of engagement' refers to what participants can do online, and what personal information is collected. Specifically we need to consider registration, moderation, and the two related aspects of data collection: child safety and data protection.
3.22 Considering registration first, there are a number of factors that may conflict:-
(a) Child protection on the Internet: there are limits on what young people with public Internet access should have been advised not to give any contact details in discussion forums. In any case they will not necessarily have email addresses to give as contact details.
(b) Data Protection: any personal data recorded should be only what is necessary for the purpose. It is normal practice for responses to consultations to name the organisation a respondent is affiliated to, if they are responding on behalf of one. However where individual young people are responding we need to justify recording anything other than the demographic data needed to associate their response with a target stakeholder group. The data protection requirements are less stringent if data does not identify individuals with their views. This could be addressed by ensuring that data requested at registration is insufficient to identify them as individuals ( e.g. "Flora, Skye").
(c) The need to know who has been included: in the engagement process there is a need to establish whether representative groups of young people have been engaged, in particular to collect data that identifies the geographical area the participant lives or works in.
(d) The need to restrict who can do what: some form of unique identifier ( e.g. password & email address) would be needed to recognise repeat use by the same individual, e.g. to authenticate that the same person is not voting twice in a poll (if the results are to be considered accurate), and there would be no guarantee that only young people were registered unless passwords and IDs were distributed through schools or youth organisations.
3.23 Taking into account the child safety and data protection requirements, together with the need to know who has been included and to restrict who can do what: -
- When contributing to an online discussion, participants should be asked to state their place (home, school etc.), and geographical area (nearest town or city).
- A registration facility would allow participants who join an online discussion to disclose the above details and their name, age & contact details once only, so they may be notified (preferably by email) about further consultation events.
- This should not however be a requirement for participation. Rather, visitors to the site who want to contribute a comment should be asked for a user-id and password if they have previously registered, or than each time they visit the site. if they were given previously in a registration form.
- Requesting the participants' gender and ethnic group could also be considered, to allow the possibility of analysing whether responses are representative and inclusive.
- Personal data requested from organised groups, for participants in face-to-face facilitated engagement events, should be only what is required to identify the minimal demographic characteristics of stakeholder groups (age group and nearest town/city, and recorded opinions should be identified only with 'screen names' (pseudonyms) unless explicit consent has been given to use the participant's name.
3.24 Scottish Executive guidance on child safety on the Internet 8 emphasises the role of schools and local authorities in minimising and effectively managing risk, and that technical options should be assessed against their value. This applies to the discussion moderation (or editorial policy) as follows: -
- Pre-moderated: this means all responses contributed by participants are vetted before they appear online to ensure conditions of use are met, and that young people do not disclose contact details for themselves or other named individuals. The risk is that young people are discouraged from contributing since they do not see their contribution immediately. The value is the greater degree of control over what is disclosed in online discussion, although it is impossible to completely ensure that individuals cannot be identified from what they say by others who know them.
- Post-moderated: this means that all responses contributed by participants are vetted within a defined period ( e.g. 24 hours). The risk is that young people might mention personal details that get used for malicious purposes before the contribution is removed. The value is that they can immediately see that their contribution has been included in the online dialogue.
3.25 In past projects we have used post-moderation on the grounds that young people value seeing their comment appear online immediately, and the risk is small if all contributions are assessed within 24 hours and the conditions of use are made explicit 9 .
Duration & sustainability
3.26 The 'duration and sustainability' refers to the period of time made available for the engagement process, and any relation to a continuing project or programme.
3.27 As we already mentioned under "Stage in Decision Making", the engagement process is likely to involve consultation on closely related issues that relate to different policy areas (or consultation owners) at different stages in the policy life cycle. As a hypothetical example, the Executive might want to consult on the risks and values associated with a selected set of RWM options, at the same time as it wishes to have a more open 'agenda setting' consultation on the establishment of the Nuclear Decommissioning Authority. Whether or not these are presented on different web sites, the 'owners' (the Executive, or CoRWM) would want to use the same tools to manage the associated information and responses to it. The stakeholders are also likely to be similar.
3.28 The ongoing nature of the engagement process indicates a need to provide structures and incentives for individuals to stay involved. In particular: -
- Participants may want to be kept informed through electronic notification, about developments in several consultations that are at different stages ( e.g. receive an invitation to take part in a debate in one consultation, together with feedback on the outcomes of another).
- The need for notification mechanisms presupposes a motivation to take part in successive consultations that are very closely related, at least in as much as they affect the general public, if not other stakeholders.
- So there is a need to consider mechanisms that will motivate continued involvement (panels, learning clubs etc.) to avoid 'consultation fatigue' among those most interested/affected by RWM issues.
3.29 Accessibility refers to measures to ensure that resources can realistically be accessed, and to assess how many citizens or other stakeholder groups participated and from where.
3.30 The need for accessibility means that a number of factors need assessed. We discuss the following considerations in more detail below: -
(a) Can young people with physical or cognitive disabilities use the tools and read the materials?
(b) Can the young people (or stakeholder groups) understand the language used?
(c) Can the young people (and other stakeholder groups) realistically use the tools and materials to meet their interests and needs?
(d) Is the take-up acceptable in terms of number of people accessing the site, and its appeal to them?
(a) Can young people with physical or cognitive disabilities use the tools and read the materials?
3.31 The Web Accessibility Initiative Content Accessibility Guidelines provide lists of 'checkpoints' or recommendations for content developers, with 3 priority levels 10 . UK Government policy is for all UK government sites to meet priority 1 checkpoints 11 .
3.32 Other important considerations here are that:-
- Young people with particular disabilities may be unable to use video, or audio, or graphical navigation techniques used in games. A text equivalent should always be available for any important information or function. This is in any case a 'priority 1' checkpoint but should be highlighted in the specification, because of the emphasis given to graphical and multimedia interaction formats in young people's web content.
- The web-based tools and materials should be accessible without the necessity to install software, e.g. as an 'add-in' to the users' web browser.
- The tools should work on both PC and Apple Macintosh platforms, and the recommended specification should be within the range typically installed in Scottish schools (if it is desirable that the tools and materials are accessible from school computers, whether or not classroom teaching and learning is to be supported).
(b) Can the stakeholder groups understand the language used in the materials?
3.33 It is considered good practice to provide content that meets different levels of ability and caters for different interests. The three sites published by the US Environmental Protection Agency (see case example in Chapter 2) for three age groups are a good illustration of this.
3.34 The ERAD/ engagement process targets 14-21 year olds, and this age range covers a wide range of literacy and interest levels. Alternative strategies for addressing the differences are:-
a) Write different materials for the different age/ability ranges;
b) Write for a reading age of 9-12, and provide links to other more detailed materials that assume a more literate audience.
3.35 Writing for a reading age of 9-12 is likely to be more appropriate. Scottish Natural Heritage guidelines for writing environmental information 12 are to write for a 9-12 reading age, since that is the average reading age of the adult population. Also the online resources in the engagement process are likely to include materials that stakeholder groups use to support their various positions, i.e. where we cannot guarantee that any measures have been taken to simplify the language used.
3.36 Translation of materials to minority languages is desirable but we should not assume that it would always be feasible to provide online materials in translated form.
(c) Can the stakeholder groups realistically use the tools and materials to meet their interests and needs?
3.37 This question is concerned with the range of activities that each stakeholder group is likely to use the tools for, and how the tools & materials meet their needs in the circumstances that they would use them in ( e.g. finding material for a school project, from home). This is what usability testing addresses. The 'usability' of software overlaps with the ease-of-understanding of the content, but should be assessed separately partly because it is a wider question and partly because content and tool are normally the responsibility of different specialists.
3.38 We make our assumptions about the users' needs and interests explicit later in the scenarios. In more general terms, the UK Government Quality Framework for UK Government Website Design (Cabinet Office, 2003) summarises the following usability issues: -
- Download delay: speed of access and speed of display between pages.
- Navigation: the arrangement of a site's navigation, the sequence of links and how they are laid out.
- Content: the amount and variety of a site's content, its word count and overall quality
- Interactivity: how well, if at all, users can perform functions on the site.
- Responsiveness: i.e. if users have a question about the site or the service it offers, how quickly and effectively can they get an answer?
(d) Is the take-up acceptable in terms of number of people accessing the site, and its appeal to them?
3.39 Take-up and appeal may be addressed by applying quantitative measures or 'web metrics' to online consultation tools. However unlike in face-to-face engagement, the individual person accessing consultation tools may not be identifiable with their actions.
3.40 While it is desirable to identify the number of participants, it is at least as important to identify the number of responses. There is not a one-to-one correspondence between an individual and their response, since in practice it is possible for one individual to respond in their own name, and/or on behalf of an organisation. Individuals can in principle also respond more than once, e.g. at a public meeting and/or in writing.
3.41 In effect the conventional approach to measuring participation is similar to the approach taken to monitoring the number of people accessing a web site, the most common measures of which are page requests and visits. These measure access to information, while responses are measured in terms of numbers of discussion "threads" and comments.
3.42 Quantitative measures of the 'appeal' of a website or tool are based on the ratio of response levels to access levels. These measures are of behaviour rather than opinion. Opinion can also be assessed, through satisfaction ratings given by users (and non-users if an offline questionnaire is used).
Resources and promotion
3.43 This refers to the costs and the approach to publicity. The 'case examples' of recent practice indicate that online engagement should be promoted alongside offline exploration of the issues (with the support of schools, youth groups). Participation that is entirely online appears to be harder to attract and sustain.
3.44 Effective means of promotion identified in the literature and practice review were: -
- Contact list, to notify contacts by e-mail
- Webrings i.e. reciprocal agreements with owners of related web sites to link them
- Fellowship programs such as those offered to undergraduates by the U.S. Environmental Protection Agency
- Competitions and quizzes
- Clickable banner ads.
Evaluation and outcomes
3.45 This refers to the approach taken to assessing the results, and how these influence the outcomes.
3.46 Evaluation criteria have already been implied in the discussion of accessibility above. Criteria for web accessibility need to be considered alongside those for effective involvement of young people. The report to the Scottish Parliament's Education Culture and Sport Committee " Improving Consultation with Children and Young People in Relevant Aspects of Policy-Making and Legislation in Scotland" (Borland et al, 2001) outlines the following principles that can also be used as the basis for criteria: -
- Accuracy in reflecting participants' views and experiences
- How representative participants are of the population
- Inclusion of groups who may not find it easy to express views
- Enjoyment by the participants
- Active engagement of the participants in decision-making
- Contribution to participants' personal development
3.47 The first two points above may be assessed from the use made of the tools, i.e. from usage data that web servers log, and from a database of the responses collected online.
3.48 The third point, inclusion, may be assessed from user registration data, indicating age group, geographical location, and possibly gender and ethnic origin. However it is also about the usability of the tools, since poor usability would exclude people without the skills to work round any difficulties in understanding how to have a say online. Usability may partly be assessed by asking users to rate the ease of use of tools.
3.49 The remaining points; i.e. enjoyment, active engagement and personal development, may be assessed from user satisfaction ratings.
Critical success factors
3.50 By 'critical success factors' we mean political, legal, cultural, economic, technological circumstances contributing to the results. Key factors to emerge from the literature and practice review that do not fit neatly into previous categories include the following:-
- Visible involvement of policy-makers and other stakeholders: Young people appear to be attracted to the online presence of experts and decision makers ( e.g. Ministers), so that they can respond directly to questions. Also a key finding of the MRWS consultation was "the need for policy-makers to be seen to be listening" 13 .
- Engagement encouraged through peer partnerships: Some initiatives have sought to link existing groups of young people online, or develop online 'virtual communities' on environmental issues. Successful initiatives based on online dialogue between small groups emphasise the sharing of experience or stories between different communities, according to ground rules that encourage consensus.
- Trust depends on a pluralist approach: The responses to the MRWS consultation indicated that "the consultation process will only work if the information given to the public is accepted as accurate, objective and complete by all interested parties" (ibid. p.4) while also giving people "the chance to test that information, for example by helping them to seek alternative views on it, so that we can earn their confidence" (ibid. p.8). This suggests an approach based on combining:-
- A core set of factual material from sources that most stakeholders consider independent.
- Position statements reflecting a broad range of values and opinions on waste management.
- Combine different media and genres: Use of multimedia seems to increase the levels of fun and engagement. Multimedia that young people produce themselves has been used as a means to stimulate confidence and communication skills. Of the online materials used in the RISCOM II Schools Project, video interviews with experts and policy makers were the most highly rated by both school students and teachers.
The Focus Group Materials
3.51 In this section we describe the materials used in the focus groups, including the background information provided beforehand, the scenarios that were used in the groups, and the process followed to get feedback on the prototype tools.
Background information on radioactive waste
3.52 Prior to the focus groups all appropriate clearances were obtained. The young people were also provided with background information leaflets on radioactive waste, published and provided by Nirex via SEERAD. These consisted of short paragraphs on the topics and questions listed in table 3.2 below.
3.53 Each young person was given a £20 book token for participating.
Table 3.2 Background information
What is radioactivity?
How do we use radioactivity?
What are the health effects of radioactivity?
How do we protect ourselves from radiation?
Will radioactivity go away?
Managing radioactive waste
What are wastes?
- High-level (or heat-generating) waste ( HLW)
- Low-Level waste ( LLW)
- Very low-level waste ( VLW)
How is waste packaged?
- Liquid high-level wastes
- Intermediate-level waste
- Low-level waste
- Where do the wastes come from?
- Where are the wastes?
Options for long-term management
What should we do with the UK's radioactive waste?
- Surface Storage
- Disposal in a deep underground repository
- Phased disposal
- Disposal at sea
- Sub seabed disposal
- at subduction zones
- in space
- partitioning and transmutation
- in ice sheets
- exporting abroad
The position in other countries
What are the other countries doing with their radioactive wastes?
Are there organisations similar to Nirex?
What types of RWM facilities are there?
- Near-surface facilities
- Mined facilities
- Deep disposal
What about the EU applicant countries?
Do Countries co-operate?
3.54 More details of the functions of the prototype tools are given later in this chapter, but their content is described in the following table 3.3.
Table 3.3 Content of the prototype tools
Diary entries by a fictional member of CoRWM
2. Video Interviews
Interview with Rachel Western, Friends of the Earth, produced for the RISCOM II Schools Website.
3. Live panel: streamed video
Question and answer session with fictional members of an expert panel.
4. Live panel: chat room
5. Game: " KU World"
Interactive online quiz produced by NRPB, where players rate the risks posed by radioactivity relative to other threats to the population of a fictional world.
6. Game: "e-democs"
Computer game-style graphic presentation of interviews with stakeholders, background information, and questions on preferences for RWM options.
7. Discussion Board
Fictional comments on RWM topics.
8. Questionnaire/ opinion poll
Sample questions to rate the importance of RWM criteria, preferences for waste management options.
9. Issue Map
Arguments for and against some of the RWM options. The authors of this report 'mapped' issues from the background information.
The Frequently Asked Questions ( FAQ) tool
3.55 We decided not to test the FAQ tool with the focus groups. We were (and remain) sure of the value of presenting background information in this format, but reconsidered what could be achieved in the limited time available for testing each tool.
3.56 The content of the tools was intended to be illustrative, rather than to adequately meet the needs for information on RWM issues. The focus groups were concerned with how that content, the presentation style, and the functions of the tools combined to meet their purpose in an appealing way (or not). Given that the purpose of the FAQ tool is primarily to meet information needs, the results would be likely to tell us what was already apparent, i.e. that the information was insufficient. That may have been worthwhile if there was sufficient time available for focus group participants to consider what they did not know, work out what they wanted to ask, use the tool to try to find it, and assess the results. A comprehensive set of test questions would be needed to thoroughly assess the ability of the tool to meet the kinds of queries typically posed by young people.
3.57 While the utility of the FAQ presentation style is not in doubt, there are issues that remain to be addressed, namely: -
- What kinds of questions about RWM would young people of various ages, educational levels and linguistic ability want to search for?
- What is the most appropriate combination of mechanisms to answer those questions, i.e. how should question subjects be grouped so they can be easily browsed, should those groups be used in a search mechanism together with 'natural language' searching, and should these mechanisms vary for different age groups?
3.58 These points are important because the research literature presents contradictory evidence on the relative values for different age groups of keyword searching, where users type in their query using Boolean operators such as "and", "or" and "not" to combine keywords, and natural language searching where phrases may be entered without such keywords or operators. Although children have been found to prefer using natural language, older age groups appear to favour Boolean operators and/or browsing using topic categories and some studies indicate that younger groups also prefer these when they know how to use them (Bilal & Watson 1998, Bilal 2000, Large et al. 2002). This suggests combining the approaches, but working out the most appropriate combination needs more detailed trials than was feasible in this study.
3.59 For information we have included a description of the FAQ tool at the end of this chapter.
3.60 These focused on two alternative settings where young people might use the tools.
- Scenario1: "Getting Involved through an Organised Group"
- Scenario 2: "Getting Involved as Individuals"
3.61 Each scenario had a general introduction to 'set the scene' as below:-
"Getting Involved through an Organised Group"
The Story So Far…
Brian is taking part in a workshop at the Youth Advice Centre, on what should be done about radioactive waste. He lives in the town of Carfuffle, about 10 miles from a nuclear power station. Brian normally goes to a drop-in session at the Youth Advice Centre every Tuesday night just for something to do, and Danielle the youth worker asked if he would turn up to a series of 6 workshops because she knows he spends a lot of time on the Internet and is interested in science.
The workshops include young people from other towns and take place over 2 months. At the end they get paid about £150 for taking part. Seven other people about the same age (16) are there, and Brian recognises 2 or 3 of them. Brian shares a computer with Jim who he knows from school.
In the first workshop they spent some time reading about the background to radioactive waste, to answer some questions they all had. Then they had spent a week talking through the options for dealing with radioactive waste, with more technical details and a discussion about how to choose between the options. Now they want to learn more details about the organisations involved in radioactive waste management, make up their own mind and have a say on the various options.
Danielle and the group use a projector linked to a computer to show what is on the screen, so everyone can see properly. There are a few other computers for the group to share.
"Getting Involved as Individuals"
The Story So Far…
Two friends visit a web site they read about, on what should be done about radioactive waste. They live in the town of Stramash, which is nowhere near a nuclear power station, but Gemma is a member of an environmental group called Greenways that has strong anti-nuclear views.
Graham her boyfriend is not so interested, except that he heard on the news that trains carrying radioactive waste will pass through town, and he's wondering how safe that's going to be. Gemma really wants her boyfriend to join Greenways but he's not that sure. Nuclear waste is work for somebody he thinks.
Gemma had seen a poster on the notice board at the Co-op, about a website where you can have their say about radioactive waste and find out more about it. Gemma memorised the address and now that they are both using the computer in her living room she takes Graham to the site. She is still trying to get him round to her way of thinking about it. Then he might go to the Greenways meetings with her.
Now Graham has agreed to find out more on what people involved in radioactive waste management issues think, so they can make their own minds up and have a say on what should be done. They are both using the computer in Gemma's living room.
3.62 Each scenario then described in short paragraphs how the tools might be used for each of three purposes, with a separate page for each: -
- Finding out about radioactive waste management
- Deciding your own point of View
- Having your say
3.63 The "Having Your Say" page for the "Getting Involved through an Organised Group" scenario illustrates this: -
"Getting Involved through an Organised Group"
3 Having Your Say
For each Tool below, please give us a comment on whether it is:-
1. Easy enough so most people would know what to do?
2. Enjoyable enough to get people thinking through the issues?
3. Useful enough for people to get something out of it (like learning something new, doing something important)?
Tool 1 Blog?
Danielle brings up a Web log ("blog") on the projector. There are different pages for people who are involved in the radioactive waste management issues. Brian suggests looking at the one by someone on the Government- appointed CoRWM committee because he's curious about what it stands for. The blog gives them a personal view about what the committee have been doing. Brian replies to the bit about visiting Dounreay, asking when it's going to be.
Tool 2 Video interview?
Danielle shows the group a video interview on the projector. She picks it from a list on a website, of people who have been involved with radioactive waste in some way. For example some are people who work for nuclear power companies, and others are from environmental groups. The one they watch is with a campaigner from an environmental group. The interview doesn't say whether there's a local group so Brian emails the campaigner to ask.
Tool 3 Live Panel Interviews?
Danielle gets the webcam talking to the Scottish Executive, where there is a panel with people from the Government Committee, the Scottish Environmental Protection Agency who write the regulations for dealing with radioactive waste, and a campaigner from an environmental group. Brian puts his hand up when he thinks of a question and tells Danielle, who passes him the microphone. Everyone can see and hear the panel members who answer. Danielle encourages everyone to ask something.
Tool 4 Live Panel Chat?
A Chat room is set up to talk to the Scottish Executive, where there is a panel with someone from the Government Committee, the Scottish Environmental Protection Agency who apply the regulations for dealing with radioactive waste, and a campaigner from an environmental group. Brian raises his hand up when he thinks of a question and tells Danielle, who types it into the chat room, which everyone can see on the projector. She encourages everyone to ask something.
Tool 5 Game- KU World?
Danielle invites the group to try the game on the shared computers. The aim is to judge between various kinds of health risk, including from radioactivity, and get an idea of the factors the government and nuclear industry take into account. You get a score, and Brian plays against other people in the group. Their scores represent their points of view about the things they are asked to decide on in the game.
The group process
3.64 The focus group events were each structured around three phases that we describe in this section. These were: -
3.65 In each phase 2 researchers took notes to record points raised by the participants. During the 'interaction ' phase the researchers also made notes of any problems encountered as the young people tried out the prototype tools, focusing especially on any difficulties that they needed help with to resolve.
3.66 This encompassed: -
- The reasons for seeking young peoples' views and the importance of the subject to future generations.
- The project background, including a reminder of the consent form contents;
- The selection of tools that would be presented in prototype form, and how we envisaged each tool supporting participation;
- An introduction to the scenarios, the criteria that we proposed the participants should use to judge each tool, and the method of applying them (see below).
3.67 The participants shared computers to try out each of the tools to judge how well it might support the purpose envisaged for it. They wrote their comments about each on sticky notes, following the guidelines presented to them with the scenarios. These asked the participants for their views on whether each tool was: -
- Easy enough so most people would know what to do
- Enjoyable enough to get people thinking through the issues
- Useful enough for people to get something out of it (like learning something new, doing something important)
3.68 Depending on the number of people in the focus group, they formed 'teams' of up to 3 people. Each was asked to read the scenarios and use the criteria to judge how well the tools met one of the purposes i.e.: -
- Finding out about radioactive waste management
- Deciding your own point of View
- Having your say
The result was a grid for each scenario. Figure 3.1 below shows an example for the East Lothian focus group (grids varied between the groups, according to the tools they looked at).
Figure 3.1 Example of focus group comment grid template
Figure 3.2 Completed comment grid
3.69 The discussion phase followed the more conventional focus group approach, although in this case it was a relatively short (20 -30 minutes) part of the process. The researchers led an open-ended discussion of topics that applied generally to the tools, and to the contexts envisaged in the scenarios. The general questions were: -
- Which of the tools can you realistically see people using of their own accord?
- Which were best for taking part in organised groups?
- Would young people be likely to use the tools once or a number of times?
- What did you feel was most important to you about any of the tools you looked at?
3.70 Specific discussion points followed on from comments noted during the 'interaction' phase. It had been hoped to widen the discussion to issues about publicity, registration and log-in, and expectations of how the Executive would respond to young peoples' views expressed online. However there was only sufficient time in one group (East Lothian) for any wider discussion, on the kinds of personal data that young people should be expected to give when registering or logging-in.
The Prototype Tools
3.71 This section describes each of the prototype tools, showing examples of the interface for each, listing their main features ("functional characteristics") and their content characteristics.
Figure 3.3 Blog screen example
3.72 The tool uses software developed by MovableType ( http://www.movabletype.org) to present the following functions: -
(1) Archives and permalinks: Access to monthly archives of the content organised by date. This content has it own permanent URL and can be accessed directly.
(2) Add your comments option: Readers can respond to posts with their thoughts. They simply write their comments in a box and click on the "post" button.
(3) Calendar: An active calendar indicates the current date, and allows readers to access the archives.
(4) Search engine: Readers can search the site and find any post, using a keyword or wildcards.
(5) Quick access to recent entries: Recent entries link provides rapid access to the last blog posted.
(6) Mailto: It's possible to send a message to the author of a comment by clicking on its name.
(7) Template: The prototype presents very simple design. It facilitates the reading and allows quick access to information.
(8) Security and privacy: In order to post a comment, the blogger has to provide his name and his e-mail address. The software also presents a set of tools "for managing, authenticating, and approving or disapproving reader comments" (cf. movabletype' website).
3.73 The prototype presents the personal journal of Wendy Roberts, a fictional member of CoRWM. She gives her thoughts and point of view on the importance of public participation and transparency as a member of the Committee. She describes some actions taken to consult the public on RWM issues and underlines the difficulties and obstacles that government has to face.
3.74 The content follows the typical characteristics of weblogs: -
(1) Familiarity (the character addresses herself directly to the young people);
(2) Subjectivity (use of personal pronouns, expression of personal viewpoint);
(3) Short concise paragraphs;
(4) Simplicity of the content;
(5) Navigability (reader can get from one blog entry to another).
Discussion board (forum)
Figure 3.4 Discussion board welcome page
Figure 3.5 Post a comment page
3.75 Forum: The forum presents a discussion on RWM. This discussion is divided into three main questions on the relevance of RWM to young people, the options for its management, and the criteria that should be used to decide between them. Each topic provides information ('information' link) and allows user to give their opinions on those issues ('enter the discussion' link) (cf the interface of the Welcome page).
3.76 Threads options: While choosing to enter in a discussion on a specific topic, the user sees the comments posted by different users. Those comments are organised by date (inverse chronological order). Indentation and an inverted arrow indicate if a post is a reply to another post. An option "reply to this" is available.
3.77 Posts options: Users can respond to a post. They simply write their comments in the window and click on the "reply to this" button. They are then requested to type in to a box (cf. the interface of the Post a comment page) and click on the "post comment" button.
3.79 Three 'information' sections were included, based on the background material sent to the focus groups. The three short sections present information through text and images: -
1. What's it got to do with us? Provides information about storage and transportation of RW. Gives direction on contributing to a debate on the questions: "How does the storage of radioactive waste affect our lives and what might happen in the future? Does it just affect people who live near power plants or everyone in Scotland? Why should we get involved in the debate and how can we do that?"
2. What are the options for dealing with radioactive waste? Provides 10 options, based on what the experts think and what other countries are doing.
3. How should we decide between the options? Provides information about the consequences of a choice: effect of RWM on local communities, wildlife and the environment, safety for workers and the public and the cost. Gives direction in order to answer those questions: "How do the experts decide between the options and what are other countries doing? What is the most important thing to think about when deciding between the options?"
3.80 Links present optional paths the user might take through the forum. He/she can access the information section from the main pages, or from the different discussion topic pages by clicking on the "list of issues" link.
Game: " KU World"
Figure 3.6 KU World screen (© National Radiological Protection Board)
(1) Progression or competition: The player has to go through the list of activities in order to complete the game. Each time, he/she gets a score for the answer provided.
(2) Single user environment: The game supports one player at a time.
(3) Multi-user environment: The player can consult the score reached by the best players.
(4) Subscription: There is a subscription procedure to follow for each new player. Personal information such as the age, the town and the e-mail address is collected.
(5) Instruction: While entering the game, the first window gives instructions about the aim of the game, how to play and strategy tips.
3.81 KU World was produced by MediaAgency ( http://www.mediaagency.com). It formed part of the National Radiological Protection Board's contribution to Science Week 2001 ( http://www.nrpb.org).
3.82 In KU world "the aim is to keep a group of people healthy by minimising the risk they take in life. Players need to protect the group from both immediate and future harm from the activities they take part in today". The player has to judge from a list of risky activities which one appears to give the greatest risk. Each choice has to be rated from low level risk to high level risk.
3.83 If the user is not sure, he/she can refer to Aidem, a fictional character who displays 'newspaper stories' presented from alternative points of view to help make an informed assessment.
3.84 By identifying the highest risk first, the player protects the largest number of people and gets a high score. While working through the list of activities, the player accumulates points. At the end, a window shows the final score, comparing it with the player's previous best score. This window provides information and links to websites on the different kind of risks. A player can also see the highest scores reached by other players.
3.85 Multimedia and interactivity is implemented in Flash. Users interact with the system while clicking on the options proposed. There are graphics but no sound. Interaction consists of mouse pointing and clicking.
Figure 3.7 E-Democs screen example © Delib
(1) Progression or competition: the player has to go through different stages in order to successfully complete the game. These are shown horizontally along the foot of the screen shown above.
(2) Single user environment: the game supports one player at a time.
3.86 The decision-making game prototype was produced by Delib ( http://www.delib.co.uk) and the New Economics Foundation specifically to support this study. The aims was to demonstrate how the DEMOCS card game could be replicated online and be supported by multimedia.
3.87 This game is based on looking at the options for dealing with radioactive waste and is divided into stages: -
(1) Opinion: player is invited to choose between options on "what we should do with nuclear waste".
(2) Information: information is provided about the options, facts about radioactivity are then listed, and the player can opt to view a (fictional) video interview with a young person about how he thinks decisions on a radioactive waste site will affect him.
(3) Scoring: in the third stage, a player is invited to rate each option proposed for managing radioactive waste.
(4) Reflection: The player has then to decide which things about managing RW appear to be more important than others.
(5) Closure: A conclusion message indicates to the player that his/her view on RWM will be communicated to the Government and that he/she will be e-mailed with updates on future decisions made.
(6) Multimedia and interactivity: A player interacts with the system by clicking on the options proposed. At the beginning, the player is invited to select the colour of a character.
(7) Fantasy: A simple character interacts with the player and invites him to answer the different questions.
(8) Curiosity: The aim of the game is to "better understand the issues around radioactive waste and to express an opinion on what should be done with it". It has the potential to provide concise information at each step.
Figure 3.8 Issue map screen example
(1) Issue map: The prototype used the Compendium system 14 to display the map. The software is produced by the Compendium Institute.
(2) Matrix of questions, ideas, and arguments.
(3) Questions - Are represented with the question mark icon ( e.g. "How do we use radioactivity?")
(4) Ideas - Are represented with the light bulb icon ( e.g. "Send it up into space")
(5) Arguments - Are represented with the + (pro) and - (con) icon ( e.g. "Would remove it from our atmosphere")
(6) Root question. The centre of the map presents the question "What should we do with UK's radioactive waste?"
(7) Node expansion: Young people were free to explore the map and click on the different issues. Users were able to consult the information simply by clicking on the icon.
(8) Facilitation tools: The prototype was designed to support offline discussion in a group setting ( i.e. with a facilitator). The aim was to help the young people to explore different aspects of the options in a graphical way, and help them to get enough information to inform their views.
(9) Editing: Users were able to add an opinion by placing the mouse pointer on an idea and clicking on the right mouse button. They could then edit their opinion.
(10) Reference nodes: Images, Microsoft PowerPoint slides and Adobe pdf format documents were embedded in the map in order to provide further information to support the decision-making process.
(1) Adaptation. Information can be visualised in different ways (small pieces of information, longer scientific documents, geographical maps and images), which allows the users with different levels of knowledge on the topic of RWM to be equally informed.
(2) Objectivity. The multiple perspectives and options available are presented in a format that is intended to present them with equal weight.
(3) Concise. The map provides a concise overview of the issues affecting the storage of RWM.
(4) Structure. The different lines of the debate are visually presented .
(5) Information Retrieval. The map aims to provide a mechanism for easy access to a large amount of information.
Live panel chat room
Figure 3.9 Chat room screen mock-up
3.88 The interface is a 'mock-up', i.e. it illustrates the features that a working version would have but does not actually provide them. It shows a chat room with its chat window where participants can type their message and read what others are saying. Two awareness options are available, a buddy list that indicates who's currently in the room and text lines to indicates the actions of the participants.
3.89 The panel discuss a question put by a young peoples' group facilitator in the chat room. Note that a log-in screen is not provided, but the assumption is that participation in the chat room would be restricted to the panel members and the facilitators of more than one group, e.g. several schools or youth groups.
Live panel by video link
Figure 3.10 Live panel: streamed video link screen mock-up
3.90 The interface is a 'mock-up', i.e. it illustrates the features that a working version would have but does not actually provide them. The fictional panel answers questions over live video links, captured using a webcam, and distributed over the Internet using videoconferencing software to present the video stream alongside a shared window (used in the example to present the web site).
3.91 It presents a fictional discussion in which a member of a specialist panel is asked to answer a question about radioactive waste management. The question is put by a moderator who sits with the panel members and reads questions that have previously been communicated by group facilitators ( e.g. teachers, youth group leaders).
3.92 The format is intended to be similar to BBC "Question Time" (see http://news.bbc.co.uk/1/hi/programmes/question_time/) an interactive question panel that allows the public to take part in a debate during its live TV presentation. Through an Internet form, via interactive digital TV or SMS, users can post comments during the programme and view other's comments. Some comments are selected by the panel moderator and presented to the public and the panel.
Figure 3.11 Online questionnaire screen example
3.93 The prototype was developed using the survey design tool www.surveymonkey.com .
(1) Structure: The questionnaire is divided into three sections: -
- Information provision: Young people are invited to first read some background information in order to answer the questions. This is the same as presented with the discussion board (q.v.).
- Questions asking opinions about RWM: The questionnaire presents two questions: "Why does it matter what happens to the waste?; How important are the things listed below when it comes to deciding what to do with nuclear waste?" The participant has to click boxes in order to answer these questions. Open question dialogue boxes are provided to allow the participant to add an open-ended comment if desired.
- Participant's profile. Several questions on the respondent's profile (age, gender, postcode, health problem or disability, occupation, use of Internet) are asked.
(2) Question formats:
- Multiple choice question
- Open comment;
- Rating + NA ("I don't know");
(3) Data validation to ensure questions are answered: Questions can be defined as optional or not, in which case if a participant does not answer a pop-up window indicates that he/she cannot go any further without doing so.
(4) Simplicity. No technical or specialised words are used in the questionnaire and the information section. Sentences are short, the writing style is conversational and questions are addressed directly to the young people.
(5) Concise. Questions are asked one at a time, and structured to request short and quick answers.
(6) Objectivity. Questions and response options are intended to be unbiased and to span the likely range of opinion, providing the alternative of an open-ended comment for views that may otherwise not be represented.
Video interview (pre-recorded)
Figure 3.12 Video interview screen example
3.94 The example used was one of 3 originally provided online and on CDROM as part of the RISCOM II Schools project.
3.95 Each question asked in the interview is transcribed and listed alongside a still image from an edited clip of the interviewee speaking. The user can click on the image to view the video, provided that the web browser is configured to automatically run 'media player' software (typically Microsoft Media Player, or Apple Quicktime).
3.96 The prototype also provided a final question "Anything else you want to ask?" with an email link inviting young people to send their own questions to the organisers, to forward to the interviewee.
3.97 The interviewee is a specialist from an environmental NGO, and the prototype was presented on the basis that a range of specialists from these and other stakeholders, including young people themselves, would be interviewed to set out their views on RWM issues and their role in waste management, or policy making on matters related to it.
Frequently Asked Questions
Figure 3.13 FAQ screen examples
3.98 This prototype is based on the Answer Tree software developed in the EDEN (Electronic Democracy European Network) project, through collaboration between ITC and European partners (Whyte and Macintosh, 2003c). It provides these functions: -
(1) Browsing: A 'tree' of questions and answers organised in a hierarchical list can be read by clicking on a question to follow the link to its answer. Question-answer pairs can be grouped according to topics (or broader questions). These are indicated in the list by a 'folder' icon.
(2) Natural Language Search: The software allows a 'tree' of questions and answers to be searched using 'natural language' i.e. without the user needing to know the appropriate keywords that represent what they want to find out, or how to combine these using Boolean search terms ( e.g. "((radioactive AND NOT nuclear) NEAR waste)". This is significant because the need for participants to have skills in searching is an undesirable barrier to their informed participation.
3.99 The sequence in Figure 3.12 illustrates how 'natural language' questions can be used to search Answer Tree.
(3) Satisfaction rating. Users are able to rate the relevance of the results they get from searching the tree. They may also send a comment to the administrator of the tree if they cannot find a satisfactory answer.
(4) External links: an answer may contain one or more links to other sites/pages.
3.100 The text follows the question-answer convention.