The Scottish Executive Social Research Unit 2005
Sue Granville, George Street Research Limited
ISBN 0 7559 1298 5 (Web only publication)
This document is also available in pdf format (216k)
Evaluation of the Masterclass Initiative - September 2005
The Masterclass programme was launched in September 2002 to support local authorities and schools in the development of ICT integration strategies. The principal aims of Masterclass are to establish a shared vision of the potential and challenges of ICT in learning at all levels and to support change in the use of ICT in learning, teaching and management.
In March 2004, George Street Research was commissioned to conduct an evaluation of the Masterclass Initiative. In order to ascertain real change in attitudes and usage of Masterclass, and to measure levels of activity in the Masterclass community, this evaluation was conducted in two phases - the first between April and July 2004 and the second between April and July 2005. This document provides a summary of the key findings of both phases of this evaluation.
Overall Impact of Masterclass
The positive impacts attributed, in part or in whole, to Masterclass in 2004 have continued in 2005 and, across most dimensions assessed in this evaluation, there has been positive movement. Given the range of ICT based initiatives in schools and the emphasis placed on this in the curriculum, it is often difficult to assess whether a positive impact in relation to ICT is attributable solely to Masterclass . However, both the qualitative and quantitative findings show that Masterclass has gone some way to achieving the aims set out in 2002.
- Masterclass appears to have had a catalytic effect on the use of ICT, although there are a number of other ICT initiatives and strategies in place that are also working to the same goal
- Over the two years of this evaluation, a key impact has been an increase in the introduction and use of new ICT equipment and this is at least, in part, associated with Masterclass
- Specific examples of good practice were provided by Masterclassers in 2004 and 2005. In 2004, these focused primarily on specific pieces of ICT and the benefit that each can bring to a school. In 2005, not only were a wider variety of uses of ICT demonstrated, but Masterclassers also provided examples of schools and local authorities taking a more holistic approach to using ICT in the school or local authority setting.
- The benefits of Masterclass to individual participants include improvements in ICT skills, opportunities for networking, sharing of good practice and championing ICT in schools and local authorities. In 2005, there has been an increase in networking across different local authorities.
- Most support elements of Masterclass (such as initial and follow-on training or the SETT conference) are well regarded. However, usage of the on-line community and facilitators is relatively limited, albeit that there has been an increase in the use of both from 2004 to 2005.
- Co-ordinators actively support the activities of Masterclassers, although time and funding constraints limit some of these activities.
- There is an ongoing need for fresh impetus to be given to Masterclass and there are a number of ways in which this can be created. These include regular meetings, increased opportunities for networking, setting up of interest groups both within and across local authorities, ongoing development of the website and pro-active communication about this from facilitators, and ongoing dissemination of good practice.
The Masterclass programme was launched in September 2002 to support local authorities and schools in the development of their ICT integration strategies. Learning and Teaching Scotland is responsible for the delivery and management of the Masterclass programme.
Currently there are around 650 participants who are involved in the Masterclass programme. Participants have been selected across all Scottish local authorities and all are key staff with a role in the development of ICT nationally and / or locally. Participants include classroom based staff, centrally based staff and senior management in schools.
Each local authority has also appointed a co-ordinator to work alongside participants, the education authorities, teacher education institutes and the Scottish Library and Information Council.
The principal aims of Masterclass are to :
- Establish a shared vision of the potential and challenges of ICT in learning at all levels
- Influence, guide and support pedagogical change using ICT across Scotland
- Provide inspiration and encourage effective leadership in the embedded use of ICT in learning, teaching and management
- Develop and share expectations of good management of ICT at all levels
- Contribute to the ongoing development of a toolkit to support the effective use of ICT at all levels
- Create a community that is able to implement and sustain the vision
- Develop both national and local capability by encouraging community members to contribute to developments at both those levels
- Facilitate the dissemination of good practice in the use of ICT across Scotland.
Aims and Objectives of the Evaluation
The specific aims of the evaluation are to :
- Assess the effectiveness of the delivery of the Masterclass programme from the perspective of all audiences currently involved in delivery, management, co-ordination and participation
- Assess the impact of the Masterclass programme on the participants themselves and on the settings in which they work
Additionally, the evaluation aims to :
- Explore the ways in which Masterclass is likely to be used in the longer term
- Ascertain the fit between the Masterclass programme and other ICT initiatives that have also been introduced (such as Leadership for Learning or Headstogether) and gain an understanding as to how ICT is used in the workplace and the benefits it can bring
- Understand the extent to which Masterclass participants are meeting the aim of becoming ICT champions for the future
- Identify any ways in which additional support should be provided by local authority co-ordinators or LT Scotland facilitators in order to maximise the effectiveness of Masterclass
- Identify examples of good practice for dissemination
The research methodology for each stage of the evaluation comprised a number of key elements. Each stage comprised initial qualitative research followed by a quantitative survey across all relevant audiences. The same respondents were contacted in both stages of this evaluation and fewer interviews were conducted in 2005 because some respondents were no longer Masterclass participants or had changed their role. The numbers of interviews completed in both stages of the evaluation are as follows :
- Qualitative research among Masterclass participants, non participants and co-ordinators within schools, Teacher Education Institutes and local authorities. In 2004, a total of 52 in-depth, face-to-face interviews were conducted across 24 organisations. In 2005, 45 of these respondents were re-interviewed within 23 organisations.
- Qualitative research among Masterclass facilitators who assist in facilitating the website by using it regularly and developing materials and activities for it. A total of 5 in-depth interviews were conducted (four by telephone and one face-to-face) in 2004 and 4 in 2005 (3 by telephone and 1 face-to-face).
- Quantitative research among Masterclass participants and local authority co-ordinators. A total of 506 semi-structured interviews were conducted in 2004 and 395 in 2005.
Effectiveness of the delivery of Masterclass
- There is a recognition that Masterclass is one of many ICT initiatives impacting on the education field at present. This makes it difficult to evaluate the effectiveness of Masterclass in isolation.
- In the opinion of Masterclassers, Masterclass appears to have had a catalytic effect on the use of ICT (76% in 2004 and 78% in 2005 agree that this is so), but this must be tempered with the knowledge that there are other initiatives and strategies in place which are working with Masterclass towards the same goal.
- In terms of the specific elements of Masterclass that have been evaluated, most of the support elements are well regarded.
- The initial training is rated very positively in terms of quality (getting an average score of 7.9 out of 10 for this). Suggested improvements centre on some segregation between groups taking part to allow like-minded individuals to learn from each other - with 40% in favour of this, and a need for opportunities to consolidate what is learned (61% want more time for this). A minority (29%) found the initial training too intensive.
- Other training elements that have been used are well regarded, especially the digital video training and learning styles training (with scores of 8.7 and 8.2 respectively awarded out of 10) although relatively few have experienced this.
- Follow-on training is also well regarded by those who have attended any sessions and participants welcome opportunities for ongoing training, either on a formal or informal basis
- A majority of Masterclassers have attended SETT, although this figure decreased from 93% in 2004 to 75% in 2005. They also regard this as a very useful event (giving it a score of 7.9 out of 10 in both 2004 and 2005).
- Only around one in two however, have made use of the on-line facility (53% in 2004 and 55% in 2005). Fewer have made use of the on-line facilitators, although this contact has increased from a level of 25% in 2004 to 34% in 2005. Levels of use of the various parts of the on-line community are limited and none of the elements of this approach global awareness, even amongst users of this facility.
- Few participants had a clear idea of what Masterclass would deliver at the outset. In the absence of detail, assumptions were made and for the largest group of participants a key expected outcome is improvement in their own ICT skills (stated by 90%). Beyond this personal benefit, however, Masterclass participants hold expectations about the nature of the community that they are joining and articulate a belief that networking (92%) and shared learning opportunities (96%) will arise.
Impact of Masterclass
- In terms of impacts, first and foremost schools have noted an increase in the introduction of new ICT equipment (and this is ongoing), although this does not always derive from Masterclass. A lot of different types of equipment are identified, but digital technology in the form of projectors, cameras and video recorders emerge as likely to have been introduced as a result of Masterclass learning, as have smartboards and whiteboards.
- Importantly, in 2005, some of the schools participating in discussions have demonstrated a more holistic approach to using ICT as well as a willingness to pilot new ICT developments.
- We have reported that Masterclassers hold expectations about improvements in their ICT skill levels, and the creation of a community of like-minded individuals that they will join. These expectations are largely realised. For example:
- Masterclass has helped improve ICT skills levels and confidence - with around 90% in 2004 and in 2005 agreeing that this is an outcome.
- A vast majority (90% in 2004 and 86% in 2005) also believe that Masterclass provides opportunities for networking although smaller proportions of all participants claim they still make use of these opportunities (76% in 2004 and 82% in 2005). Local authority co-ordinators play a key role in facilitating networking opportunities and a large majority of participants believe that they do help.
- The 2004 data showed that there was a need for networking opportunities at a cross local authority level and this has increased slightly in 2005, with a greater proportion of participants referring to networking across local authority areas (11% in 2005 compared to 0% in 2004 claim that there is a great deal of networking across local authority areas). However, this may have been at the expense of networking within local authority areas where the proportions claiming this happens a great deal has decreased from 28% in 2004 to 15% in 2005.
- Some 88% in 2004 and 78% in 2005 of participants have had a chance through Masterclass to share good practice and opportunities for this are still considered to be increasing, albeit that a smaller proportion claim this is the case in 2005 (50%) than in 2004 (65%).
- While loathe to regard themselves as ICT champions, Masterclassers are active in a wide range of champion activity, albeit that this is still more evident within schools than within their own local authority area or outwith the area. Masterclassers are most likely to say they have had an impact on the use of equipment within schools and to have improved knowledge about ICT but some are also influential on policy and developing best practice.
Level of Activity
- Overall, there is a feeling that fresh impetus needs to continue to be given to Masterclass.
- Meetings for Masterclassers are held on average every two months, but this does seem to vary by area. This has not changed across the two years of this evaluation.
- Just over one in two in both years of the evaluation have made use of the Masterclass on-line facility. Even among those who do use the facility, the frequency of use is low overall and this has only increased slightly in 2005 (an average of 19 times per year compared to an average of 17 times per year in 2004).
- Co-ordinators actively support the activities of Masterclassers in a number of ways which are essential to the development of networking opportunities, the sharing of good practice and the creation of ICT champions. However, the 2004 survey showed that co-ordinators face resource constraints and acknowledge that there are ways in which they would like to offer more assistance, and this has continued in 2005. Ongoing support in terms of resources and time is needed, alongside support from Learning + Teaching Scotland.
Barriers and suggested improvements
- There is no widespread belief amongst Masterclassers that the selection process to find Masterclassers should be altered, but some believe it would be better if it is more open or that more information is made available so that volunteers can come forward.
- In terms of selecting candidates, one aspect to consider is the extent to which staff have time to disseminate their Masterclass learning to others. Those who have time ring fenced for ICT development activities within their current school role are more likely to be able to fulfil this aspect of activity.
- A majority of Masterclassers (83% in 2004 and 87% in 2005) feel that they would like others to have a chance to participate in Masterclass.
- There is a need for continued ongoing support with 48% in 2004 and 53% in 2005 saying it is hard to know how to put into practice what you learn and just over 70% in both years favouring more meetings to help them.
- Meetings play a key role in allowing networking and sharing of good practice but occur with varying degrees of regularity.
- In terms of the on-line resource, the high level of non-use identified in 2004 has continued in 2005. Awareness of the re-launch of the on-line resource was high among those participating in qualitative research and the proportion of those experiencing access problems to the on-line community has decreased from 45% in 2004 to 19% in 2005. Co-ordinators need to continually encourage participants to use this. Among users of the on-line community, there is a need for greater awareness of what is available within it, for it to be kept up-to-date and to ensure that the needs of different groups are catered for.
- There is still some demand for more opportunities to network, although the proportion feeling that networking opportunities could be improved has decreased from 72% in 2004 to 61% in 2005. The data also shows a decline in the proportions of those who say they could share good practice more (60% in 2004 and 52% in 2005). The channels for doing this are similar in each instance and more meetings, a better on-line community and the provision of cover or time set aside to allow Masterclassers to do this will facilitate this activity.
- In both years of the study, around one third stated that they would like opportunities to meet others in their peer group, although this figure has increased by 5% in 2005.
The recommendations arising from this evaluation centre on ways to improve the role of the Masterclassers, and improving the support developed in relation to Masterclass and from local authority co-ordinators and at a national level. A number of the recommendations made in 2004 have been implemented and the data shows that these have already had a positive impact in many instances. However, in 2005, the same key recommendations emerge.
Improving the role of ICT champions
- There is a tendency for champion activity to be dismissed in conversation, arising largely from a dislike of the champion term.
- A beneficial form of support to Masterclassers is making key speakers available. Masterclassers would welcome a planned programme of talks by speakers such as Alan November.
- There is a need to ensure that the initial motivation and enthusiasm for Masterclass is maintained amongst Masterclassers. The on-line facility has been relaunched at the 2004 SETT conference, although there has only been a slight increase in usage as a result of this. There is a need for this to be given fresh impetus on a regular basis and Masterclassers should be encouraged, in fairly rigorous terms, to access the facility on a regular basis.
- Further support for the status of Masterclass should be provided at every opportunity, through the SETT conference and by publicity and write ups of good practice arising from Masterclass. This should be made available as widely as possible with attention drawn to the presence of Masterclassers in schools or the education department.
- Learning from Masterclass should be made more easily accessible to others with an interest in ICT. Champions could be encouraged to involve others within their school in ICT development and local authority staff could encourage this by setting goals.
Improving support related to Masterclass
- Some local authorities still appear to be struggling with the impact of Masterclass commitments on the other duties of co-ordinators, although some have involved other local authority staff in development of ICT initiatives in order to help keep impetus going.
- The on-line facility needs to be kept up to date with fresh innovative thinking to ensure that it is regarded as a key facility to be used on a regular basis.
- Any training of new Masterclass participants also needs to make sure that new Masterclassers realise the importance of the on-line community.
- Undoubtedly, the maximum impact from Masterclass is found where Masterclassers have been able to make use of the learning provided. The development of some exercises to encourage Masterclassers to consolidate their learning from the training would be helpful.
- The introduction of a mentor system at a local level could provide fresh impetus for Masterclassers and support to local authority co-ordinators.
- There needs to be continued provision of a newsletter, providing updates and examples of good practice.
Improving the support from local authorities
- The role of local authority co-ordinators in impacting on the outcome of Masterclass is significant. The study indicates that this aspect needs to be adequately resourced in terms of :
- budgets for funding Masterclass driven initiatives
- integration of Masterclassers into strategic development and decision making within schools / local authorities
- provision of support for champion activity through regular Masterclass meetings and other opportunities for sharing such as conferences, exhibitions and joint training
- Some guidance arising from this study might usefully be developed for local authority co-ordinators indicating the mix of very effective roles that they can provide in support of their Masterclassers.
- Additionally, co-ordinators might consider ways of devolving responsibility for some support activities to others within their area.
Improving support at a national level
- One of the main gaps at present continues to be in the cross-pollination of ideas, learning and knowledge between local authorities. LTS could consider ways in which it might provide mechanisms for ICT co-ordinators to exchange learning and ideas through the creation of local authority peer groups, for example.
- Consideration might be given to ways of developing a pool of subject and sector champions, to reduce the feelings of isolation experienced by some individuals.
If you wish for further copies of this summary document or have any enquires about Masterclass, please contact Marie Dougan at L T Scotland ( M.Dougan@LTScotland.org.uk).
A full copy of this report is available at < SEED to insert URL>