Key Findings from the National Evaluation of the
New Community Schools Pilot Programme in Scotland
This document is also available in pdf format (135k)
Welcome to Insight
Insight is a publication of the Research, Economic and Corporate Strategy (RECS) Unit, one of four units in the Information, Analysis and Communication Division, which is responsible for providing analytical services within the Scottish Executive Education Department (SEED). RECS is a multidisciplinary unit (consisting of researchers and economists) which undertakes and funds economic analysis and social research in the fields of: school education; children, young people and social work; architecture; and tourism, culture and sport.
The Scottish Executive is committed to the use of sound evidence in the development of policy and practice as well as in the evaluation of policy and its implementation. We therefore want to disseminate the results of research that SEED has undertaken and funded, in a manner that is accessible, interesting and attractive.
Insight aims to present the essence of research projects in a format that will be useful and informative for practitioners, policy makers, parents, academics, and anyone else who has an interest in economic and social research in these areas.
Key Findings from the National Evaluation of the New Community Schools Pilot Programme in Scotland
Pamela Sammons, Sally Power, Karen Elliot, Pamela Robertson, Carol Campbell and Geoff Whitty (Institute of Education, University of London)
' The pilot programme of New Community Schools will be concentrated in disadvantaged areas where children face significant risk of social exclusion and formidable barriers to learning in their everyday environment.' (Scottish Office, 1998; 5)
The New Community Schools (NCS) policy is an integral part of the Scottish Executive's wider Social Inclusion Strategy. In recognition of the relationships between educational achievement, health and socio-economic factors, NCSs were charged with expanding and integrating the range of services offered to young people in disadvantaged areas with the intention of both raising attainment and promoting social inclusion. The NCS Prospectus (Scottish Office, 1998) outlined five key goals:
- modernisation of schools and the promotion of social inclusion
- increasing the attainment of young people facing 'the destructive cycle of underachievement'
- early intervention to address barriers to learning and maximise potential
- meeting the needs of every child, ensuring that services are focused through NCSs
- raising parental and family expectations and participation in their children's education.
The 37 NCS projects in the Phase 1 pilot programme involved over 170 schools or institutions in 30 education authorities. Some projects were single schools but most comprised clusters of locally associated schools, with funding in the form of a specific grant from The Scottish Office Excellence Fund to enable NCSs to operate in an integrated way. The cost of service delivery was to be drawn principally from existing education, social work and health programmes. The majority of Phase 1 pilot projects commenced in April 1999 and ran until March 2002. SEED commissioned a national evaluation of the Phase 1 pilot in April 2000. More projects were introduced in Phases 2 and 3 in 2000 and 2001 respectively and a programme of roll out 1 across all schools in Scotland began in 2002.
The national evaluation: context, aims and methods
The national evaluation explored the extent to which the 37 pilot projects achieved the 'essential' or 'likely' characteristics of NCSs based on the programme goals set out above. Its three main aims were:
- to inform the development of the Phase 1 NCS pilots and aid future decision making
- to provide data and analysis on key, school-based outcome measures so that a comprehensive national overview is available at regular intervals
- to provide a summative judgement on the effectiveness of NCSs and hence inform local and national policy-making about their longer-term future.
Factors that helped NCSs to be effective, as well as barriers to success, were investigated under five major themes: leadership and management; multi-agency provision; pupil experience and attainment; family engagement; and community engagement.
The national evaluation adopted a mixed methodology and drew on a range of evidence sources. Surveys were conducted with separate questionnaires at the project, secondary, primary and nursery level across the 37 pilot projects and individual schools. The surveys provided indicators of key areas of NCS activity at baseline (i.e. prior to NCS status, 1998/99), Year 1 (1999/2000) and Year 3 (2001/02) of the pilot phase. A separate survey focusing on vulnerable groups of children was administered in Year 3. 2 Six case studies of NCS pilot projects were selected to represent different management and organisational models; local contexts, such as rural or urban; size in terms of the number of schools, staff and young people involved; and focus, scope and range of activities. The case studies involved visits, interviews, observations and staff surveys. Documentary evidence was also collected. In addition, a range of school-level statistical indicators was obtained from SEED covering levels of pupil attainment, attendance and staying on rates 3. The analyses included a particular focus on cohorts of pupils with low attainment (those not achieving the minimum levels of attainment for their age) because this group is of particular interest in terms of SEED policy aims.
Leadership and management
What are the most appropriate management arrangements for facilitating multi-agency work in NCS?
The management arrangements of Phase 1 pilot NCS projects varied considerably, but the majority appointed a full-time 'integration manager' (IM) _ a designation that captures the complex tasks of managing horizontal networking across agencies and institutions while also developing appropriate line management for different professionals working within or beyond the core NCS team. Case study evidence suggests that IMs faced substantial challenges in overcoming barriers among different professionals and institutions, although many made significant advances as time progressed. Also problematic was the balance between IMs' line management responsibilities for NCS core staff members and their management relationship to other senior agency partners, including headteachers. In some case studies, these management arrangements lacked clarity and were at times difficult in practice. Overall, the case study and survey findings indicate the crucial importance of an IM or equivalent with dedicated management responsibilities for the early development and successes of NCS projects.
The survey also indicated that overly complex or bureaucratic arrangements involving multiple and overlapping management layers hindered progress. More streamlined and strategic management arrangements combining senior level support and a range of relevant partners contributed to NCS achievements. In general, the relationship between NCS governance and other existing statutory and voluntary multi-agency arrangements benefited from review and strategies to co-ordinate and simplify approaches.
The quality and frequency of liaison and communication between professionals was variable across and within NCSs. The survey indicates that formal and informal meetings between NCS core team members were valued to ensure good communication. Although more meetings may have helped to address some remaining communications issues, the case study findings indicated the need to balance time spent for meetings with action to initiate and implement activities that directly impacted on NCS delivery. The co-location of NCS core staff generally contributed to ease of communication and liaison.
All pilot NCSs were expected to undertake local evaluation of their project. Documentary evidence indicates that formalised local evaluation was under-developed in most projects particularly in set up and Year 1. Later survey evidence indicates some improvement _ with new or enhanced forms of monitoring, review and evaluation being promoted (and NCS involvement seen to be making a 'moderate' or a 'considerable' contribution in two fifths and over a third of NCS projects respectively). However, formal systems for monitoring and evaluating the progress of the pilot as a whole were in place in only a minority of cases. More commonly, local evaluations focused on isolated areas of activity, such as parents' and pupils' use, and views, of breakfast clubs or the implementation and impact of Personal Learning Plans (PLPs).
Multi agency provision
The majority of NCSs reported some set up problems in Year 1. Difficulties in getting started were rated as 'considerable' by one-third of survey respondents and 'moderate' by a further quarter. Reported issues frequently related to funding, appointing staff, setting up appropriate procedures, accommodation issues and lack of shared clarity about the purpose and working of the NCS. While pilot NCSs sought to address these issues over time, initial difficulties slowed the implementation of initiatives in some projects.
What strategies can be developed by senior managers in different services to ensure that mainstream funding supports the growth of NCS multi-agency work?
NCSs were intended to draw in funding from both social work and health sources to match funding from education, although there was some lack of clarity about procedures and requirements for this. The Year 1 survey indicates that half of Phase 1 pilot NCSs reported they had received matched funding from both health and social work sources. By Year 3, the majority of these NCS projects indicated the receipt of funding from health sources and just under two-thirds reported this from social work. In addition, NCS projects received a variety of 'in kind' funding which mainly took the form of extra staffing. The Year 3 survey indicates that, where received, additional resources for NCSs were perceived to be a major factor contributing to NCS successes.
The remit and range of staff appointed to support NCS activities varied depending on individual projects' approaches and levels of resources. In general, staff were selected to cover key activities in terms of health (e.g. school nurse, health promotion officer), social welfare (e.g. social work, family worker) and education (e.g. learning support, home school liaison officer). Staff committed to the multi-agency approach of NCSs, and appointed on a long-term basis, provided considerable benefits to the NCSs in terms of expertise and capacity to deliver multi-agency services. However, the short-term funding of NCS pilots created problems of recruitment and retention, with staffing difficulties identified in the survey as hindering the development of some NCS projects. Case study evidence indicated the importance of consistent staffing to develop trust and confidence with vulnerable individuals receiving support. Responses to the vulnerable children survey indicated that additional staff, financed through NCS, contributed to improved recording and monitoring of vulnerable groups of children in schools, in addition to the swift identification of the needs of vulnerable groups.
NCS projects supported a considerable diversity of services and initiatives. The case study evidence suggests that although there were relatively few completely new initiatives, most NCSs contributed additionally through enhancing or extending existing projects and attempting to ensure that they reached those most in need. The survey findings, however, indicate that most NCSs reported that they became involved in implementing and delivering an increasing range of education, health and social policy initiatives during the three years of the pilot. The extent to which these can be seen as separate from NCS activity in some projects is unclear. The NCS initiative appeared to act as an important catalyst to promote change and supported the combination and integration of different initiatives, reflecting a stated policy intention for Phase 1 NCSs.
The survey indicates that staff, management and senior partners' commitment to the NCS concept and support for multi-agency working were seen as major factors contributing to achievements in some NCSs, while the absence of such support and commitment was a major barrier to development elsewhere. The case study findings, including staff surveys, indicated overwhelming support for the principle of multi-agency working. The vulnerable children survey showed that NCS staffing, multi-agency meetings and targeted services provided extra support for specific individuals and groups; however there were varying perceptions on the extent to which these activities benefited vulnerable children. The survey indicated that NCS assistance had benefited children at risk of exclusion in secondary schools (a third of respondents reported the impact was 'considerable', a further third 'moderate'). NCS impact varied between projects and specific pupil groups and was seen to be weakest for looked after and accommodated children in primary schools, where over two-fifths felt the NCS joint agency approach had 'none/minimal' impact, although this is likely to be due to the very small numbers of children in this group in individual primary schools (around two-fifths of primaries reported they had no children in this category on their rolls).
A range of barriers to multi-agency working had to be overcome to deliver multi-agency services. These included practical issues of different working hours, holiday arrangements and accommodation issues and professional issues of confidentiality procedures and levels of formality. Making time and creating opportunities for staff to meet together to address such matters and develop strategies for future practice were reported to have promoted multi-agency working.
The case study evidence indicates that the scope, nature and practice of multi-agency professional development varied considerably. In the Year 3 survey, half the respondents indicated 'moderate' levels while another third of respondents noted that their NCS project had been of 'considerable' influence in providing this development, a substantial improvement on the equivalent findings for the Year 1 survey. Case study and survey findings both indicate that where staff development was perceived to be effective, it supported the development of collaborative cultures and practice by removing barriers to multi-agency working. This was seen to have contributed to NCS achievements and potential sustainability in some projects. Survey evidence also indicated benefits of training to develop understanding and confidence in working with vulnerable children. Nonetheless, the overall extent to which NCS projects had contributed to multi-agency training for the specific needs of vulnerable children was reported as fairly limited.
Pupil experience and attainment
NCS strategies for raising attainment and increasing pupil engagement were primarily focused on behaviour and welfare. Case studies and survey results indicate that a wide range of strategies, including out-of-school hours provision and support programmes for pupils with particular difficulties, was adopted. Across the case studies, young people reported the positive effects of specific NCS activities on their lives and attitudes. Professionals and families interviewed also endorsed the beneficial effect of NCS programmes for the pupils involved, particularly in terms of tackling disaffection. These positive views concerned both work with small targeted groups and with larger groups. Breakfast clubs were perceived as a particular success in terms of increasing engagement and promoting health at the primary school level. The Year 3 survey showed that half of all Phase 1 primary schools and over a third of secondary schools responding had introduced a breakfast club and, of these, almost all reported that NCS involvement had led to their development.
The Year 3 surveys reported that NCS activity had improved pupils' attitudes to school. Just under half of primary and nearly two thirds of secondary respondents rated the impact of the initiative on pupils' attitudes to school as 'considerable.' Case study evidence focused on vulnerable young people in particular and revealed that the pilot programme helped improve their attitudes to education and attendance.
What approaches are needed to encourage the accommodation of NCS principles within learning and teaching in the mainstream curriculum?
The NCS impact on the curriculum was most marked in the areas of health education and promotion. Survey evidence indicates a striking increase in involvement in health initiatives in most pilot NCSs. Respondents perceived considerable progress in the extent to which they could describe their school as 'Health Promoting' with many specific new activities and initiatives. Another development particularly evident in the case studies was the provision of alternative vocationally based curricula for young people at the secondary stage. Although these involved only small numbers, they were generally viewed as successful in sustaining involvement in education.
What steps should be taken to enhance teachers' commitment and enable PLPs to be an integral and fundamental part of teaching and learning?
Taking more account of young people's views is an important feature of SEED policy and was reflected in NCS aims, guidance and other documentation. However, in some case studies certain vulnerable young people interviewed still felt excluded from wider consultation processes. Nevertheless, an increased number of schools in pilot NCSs reported the existence of a pupil council or equivalent body. By Year 3, nine out of 10 responding secondary schools had a council. Three quarters of primary schools also had councils, though the perceived influence of NCS involvement was weaker.
Personal Learning Plans (PLPs) featured strongly within the NCS policy. One third of primary schools in the pilot reported 'considerable' implementation of PLPs in the Year 3 survey, with just over half reporting that the NCS had been a particular influence in their development. Around half of secondary schools reported 'considerable' implementation of PLPs and most of these indicated that the NCS influence in their development was 'considerable'. Both sectors reported a substantial increase in PLP implementation from baseline and Year 1 figures. Written comments, however, indicated that substantial numbers of schools still perceive difficulties or have made little progress in this area. In nursery schools PLPs were still infrequently used, almost two thirds reporting 'none/minimal' implementation by Year 3. Case study evidence also suggested that PLPs were slow to get started and that there was considerable ambivalence from teachers. In some case studies, PLPs were most developed by processes outside the conventional classroom and teacher involvement was limited. This meant that they were not an integral part of pupils' classroom learning.
The surveys revealed mixed views about the perceived impact of the pilot NCS projects on pupil attainment. Almost half of both primary and secondary school respondents reported a 'moderate' positive effect. By contrast, in the secondary sector over half perceived the impact to be 'limited'. Most school-based professionals involved in the case studies believed that it was too early for the NCS programme to have had any effect on overall patterns of attainment. They felt it was unreasonable to expect a discernible impact on overall school attainment rates when most of the NCS targeted initiatives focused on relatively small numbers of vulnerable young people and their families.
What changes need to be made in NCS strategies to ensure greater positive impact on attainment and other pupil outcomes?
Involvement in the NCS pilot was intended to focus on lowest attaining schools and those with high incidence of social disadvantage. Analysis of school-based SEED 5-14 attainment data from 1998/99 to 2000/01 showed that schools in the pilot had the lowest percentage of children attaining the minimum expected level (or above) in the 5-14 assessment programme at the start of the initiative for both primary and secondary sectors. As a group, schools in the NCS pilot showed fairly steady improvement over three years. By 2000/01 (Year 2 of the pilot) more pupils reached the minimum expected attainment levels for their age (primary and secondary). However, these trends were also found in schools across Scotland and for all phases of NCS involvement. Although there is considerable variation between schools (particularly primaries) and some showed well above average levels of attainment increases, schools in the pilot did not show overall greater improvement than others. There is little evidence, therefore, that the overall attainment gap between pilot NCSs and other schools narrowed during the period 1998/99 to 2000/01. Results for Standard and Higher Grade showed less change in their attainment patterns over the three years. Again, there were improving trends for all schools in Scotland and all NCS phases.
Analyses of SEED statistical data on pupil attendance compared half-day absence rates (authorised and unauthorised) and those through temporary exclusions. The results indicate generally no significant difference between national trends and those in Phase 1 pilot projects. There was no general improvement in attendance over the period 1998/99 to 2000/01. Statistical data on staying on rates (considered as a proxy indicator of raised expectations) showed an increase in the percentage of pupils staying on to S5 and to S6 nationally. There was no evidence that, as a whole, schools in the pilot improved their relative position for this indicator; though at S5 a moderate improvement was recorded (under 3%). There was some evidence of similar modest improvement in the proportions of pupils from Phase 1 pilot schools going on to higher and further education, though the trend was not statistically significantly different from the national pattern.
Evidence from the case studies suggests that, where extra support was provided for vulnerable children and young people, it helped to keep them in mainstream schooling. This applies where there was group provision focused on particular needs, such as anger management or alternative curricula; and also to individually targeted provision, such as counselling and mediation. A major factor in success appeared to be increased capability within NCSs to identify problems and draw agencies together to provide solutions. The case study evidence indicates that improved pupil behaviour and attendance were evident for vulnerable pupils interviewed.
The identification of and provision for vulnerable children was explored by a separate survey focusing on three groups: looked after and accommodated; the lowest achieving group; and those affected by exclusion. The survey indicated that the numbers of children in the looked after and accommodated groups and exclusions were very low at the individual school level, especially in primary schools. There was some evidence of increases in the numbers of looked after/accommodated children reported by schools (probably indicating improved identification) but reductions in numbers excluded. In connection with exclusion, many pilot NCSs reported involvement in Alternatives to Exclusion and other initiatives intended to support pupils with significant behavioural difficulties.
Only a minority of primary and secondary schools reported that involvement in the Phase 1 pilot NCS had a major impact on monitoring and recording for these groups of vulnerable pupils. In part this reflects the pre-existence of such systems in some schools and the influence of other initiatives and national requirements. Nonetheless, the increase in the number of respondents able to provide information about vulnerable pupils over four years (1998/99 to 2001/02) suggests that recording systems within most schools had improved. The swift identification of the needs of vulnerable children is a priority in NCS policy and high proportions of both primary and secondary respondents perceived that this was now being achieved.
How can good practice in the identification of needs and pastoral care of vulnerable children be extended to include a clear focus on monitoring their attainment and learning?
General levels of satisfaction with current provision for vulnerable groups were moderately positive, especially in the secondary sector and most particularly in relation to the lowest achieving group. However, respondents saw the specific contribution of the NCS initiative to this as 'limited'. The NCS impact was seen to be greater for children at risk of exclusion, with nearly half of both primary and secondary respondents reporting the NCS had a 'moderate' impact on raising the attainment of this group. Furthermore, one tenth of primary respondents and one fifth of secondary respondents thought their NCS had exerted a 'considerable' impact in raising the attainment of this group.
Engaging all parents, including those of the most vulnerable pupils, was seen as a long-term task by NCS participants and did not sit easily with the three-year timescale of the NCS pilot. Year 3 surveys showed that most initiatives developed by the NCSs for families involved outreach work with parents, and by the end of the third year, the majority of NCS schools _ both primary and secondary _ had formal access to a home-school liaison officer and a social worker. The increase was attributed to NCS involvement in just under half the cases for home school officer (the rest often being funded by the Early Intervention programme). In almost all cases the availability of social workers was directly attributed to NCS status. Respondents' comments suggest that in practice access was difficult in some cases because a part time post might be serving several schools in a project. Posts were reported to be hard to fill in some cases because of the short-term nature of the appointments and responsibilities spread over many schools. The case study projects' experience of employing such staff to reach vulnerable parents was mixed. There were reported difficulties of ensuring continuity and maintaining trust in sensitive situations but evidence that individual families benefited from the services provided.
What are the most appropriate and sustainable kinds of strategies to involve all parents and carers in a community?
The case study research showed that NCS projects had variable success in promoting family engagement strategies, particularly where there was a mismatch between anticipated funding and support and amounts actually received. Although parents who had been involved in particular initiatives knew about the pilot NCS, there were concerns that general awareness of NCS projects was limited, despite efforts to promote specific NCS initiatives and the project as a whole through newsletters and local media. Nevertheless, staff felt that parents were involving themselves in more learning opportunities.
Relatively few initiatives were directed at engaging parents in dialogue about their child(ren)'s education. Where this was done, it appears to have led to increased engagement and confidence. A more common approach was the provision of services directed at parents in need (such as holiday clubs, family liaison officers). These appear to have been well received by parents, although the numbers involved were quite small.
The NCS Prospectus intended that pilot NCSs would provide an important opportunity to build the capacity of the local community. However, the survey and NCS annual reports indicated that community engagement strategies were slow to get off the ground in most Phase 1 pilot NCSs in Year 1, and it remained the least well-developed strand of the NCS pilot. In Year 1 the majority of schools in the pilot reported only 'minimal' or 'limited' activities involving the wider community. By Year 3, there was evidence of some progress, particularly by secondary schools, with over half reporting that they were now moderately or considerably involved in community activities. The case studies found few instances of systematic consultation with communities. The extent of general community awareness of NCS activities is hard to gauge, and less than one third of the pilot projects reported producing newsletters for the wider community.
What leadership and funding strategies can best promote further community participation and impact?
A range of new community activities was reported across the pilot projects including the provision of adult learning opportunities (particularly relating to IT), social and sporting events. The number of schools reporting access to a community education or youth worker quadrupled over the pilot phase. This development was attributed directly to NCS in three quarters of schools. Schools also reported increasing community use of their facilities. For primaries an increase was more evident for provision that was free, but for secondaries the increase was more notable for provision for which a charge was made. This may have implications for community accessibility.
However, while the surveys showed that positive progress was perceived in community engagement, the case studies suggested that this was generally patchy. Indeed, in one case study community engagement had not been attempted during the pilot phase, although there were plans to focus on it post-pilot, when longer-term funding had been secured. In others, however, there were examples of clear engagement. For instance, the provision of vocationally-based learning opportunities led to some vulnerable young people and adults pursuing further education courses. There were also some innovative youth work activities aimed at reducing crime and drugs use. The most successful aspect of community engagement noted by case studies was the development of greater links between schools and local voluntary and community organisations.
A key difficulty in establishing community engagement was reported to be the short-term nature of the Phase 1 pilot NCS funding, which did not facilitate the development of high trust relationships necessary to build up sustainable links between schools and communities. In several cases effective networking and linking appeared to depend largely on the commitment and attributes of the IM in post. This raises questions of whether progress in community engagement can be sustained in the longer term.
Summary of main achievements reported for the Phase 1 pilot NCSs
The Year 3 survey asked respondents to identify the main achievements of their NCS. Most projects reported a much greater emphasis on multi-agency/multidisciplinary approaches and procedures as one of their main achievements. This was seen to have increased accessibility of services to vulnerable children and families in particular. The case studies likewise pointed to the expansion of inter-professional understanding and establishing of links amongst different groups.
Increased provision and support for vulnerable children and families were cited as important achievements by over half the projects. Survey evidence showed that NCS involvement was perceived to have promoted more inclusive approaches for vulnerable pupils and to have improved the recording and identification of vulnerable groups, particularly looked after and accommodated children. The case studies likewise provided evidence that NCS involvement enhanced existing provision.
Survey responses commonly cited improved pupil outcomes as an area of achievement, although specific instances and figures were not usually reported. Some commented on a significant rise in attainment and improved attendance, others on 'dramatically reduced' exclusion rates and significant differences in post school destinations of young people. As mentioned above, improved attainment occurred in all schools in Scotland. Perceptions of whether involvement in NCSs had led to an improvement in pupils' attitudes to school were generally positive.
The survey drew attention to increased community and/or parent and family engagement as areas of reported success in over half the pilot projects. Parent education was noted as a particular success by some. Increased involvement of parents in their child's education and a raised profile of the NCS in the community were also cited as successes. The case study findings were more equivocal.
Participants highlighted a wide range of new activities, including curricular developments and out of school care/learning, as main success areas in particular projects, especially in all aspects of health. Examples included curricular developments in mental health, personal social development, alternatives to exclusion, healthy eating. Most curricular developments reported were 'add ons' or targeted for vulnerable pupils rather than reflecting change in typical classroom experience. PLPs might encourage curricular change and pupil engagement, but mostly their development and implementation proved slow and teachers were ambivalent about their value and the workload implications.
In summary, while the new activities and initiatives individually and in general should be viewed as positive developments, they point mainly to Phase 1 pilot NCS projects offering significant 'additionality' in terms of extending multi-agency involvement in service provision and targeting services to identified individuals rather than offering, as yet, radical or innovative new services to modernise schools.
Factors that contributed to achievements
Both the survey and case studies pointed to a number of common factors that appear to have contributed to significant progress within the pilot phase:
Commitment to NCS concept: Positive aspects stressed included commitment to the NCS concept and multi-agency approaches at both institutional and local authority level, with support from key individuals (political and senior management).
Support from other agencies: Projects that enjoyed support from their own education authority and from partner agencies at senior levels reported being able to implement strategies more effectively. Matched funding and resources freed staff to focus on new or enhanced activities. Positive outcomes included pastoral work in primaries, community development worker input and support from school nurse and social work.
Team working: Staff effectiveness was promoted where there was team co-location, willingness to work together, learning from one another, a shared vision, strong leadership combined with staff training and development opportunities leading to joint working practices, integrated planning and working. The IMs' commitment, experience, contacts and effectiveness as a change agent was an important contributory factor to success in some projects.
Staff development: Work oriented staff development was a particular feature in some projects and where this was accredited it was seen as an additional benefit. In written comments many respondents noted the value of 'joint learning' and the variety of staff development facilitated by their NCS. The availability of additional funding for training, supply cover, conferences, etc was viewed as a positive feature of the pilot.
Clear management structures: The survey and case studies emphasised the benefits of streamlined and efficient governance and a distinct focus on specific tasks and action. The identification of a post holder with dedicated management responsibilities (such as an IM) for the NCS benefited its swift development.
Resources: Funding for three years facilitated NCS developments and fostered autonomy and the ability to work flexibly to develop local solutions to local contexts. The attraction of other additional funds from linked initiatives was found to have been of great benefit. NCS funding combined with funding from European Social Fund, New Opportunities Fund and the Excellence Fund were cited as factors that promoted implementation and success.
Factors that hindered pilot projects
The factors reported to have hindered projects in the Year 3 survey findings were often the opposite of those identified as facilitating success:
Lack of clarity and commitment between partners: Lack of understanding of different agency roles, remit and of what NCSs really stand for and resistance to change were reported by some survey respondents as a particular hindrance. Tensions were noted between education and other staff in aims / objectives and the perceived value of different approaches for vulnerable young people.
Competing priorities: The wide range of aims of pilot NCSs was perceived to cause problems in some projects. A few respondents felt raising achievement was at variance with the inclusive aims and promotion of pupil welfare and motivation underlying the NCS concept. Stresses in partners' sectors, such as shortages in staff or impact of local authority reorganisation, were thought to have contributed to a lack of ability and willingness to invest heavily in pilot NCSs in some cases.
Overly complex management arrangements: Lack of clarity combined with overly complex managerial responsibilities within and between partner organisations created difficulties within some NCS projects and sometimes led to confusion and conflict.
Short time scale and lack of clear planning beyond the pilot: The short time scale and lack of clear planning beyond the pilot period hindered start up and sustainability in a number of projects and had a negative impact on staff recruitment and retention.
Resourcing: Where one project included many schools and / or matched funding was not achieved, the additionality of NCS resources had to be very thinly spread.
Professional barriers: Difficulties with bringing staff from different agencies together were reported, from practical issues such as scheduling meetings to differences in professional cultures. Some projects noted that their NCS was initially viewed as an education initiative by partners and that considerable effort was required to encourage joint ownership and the pooling of resources, leading to delays in start up and implementation.
Implications for policy and practice
What mechanisms need to be put in place to ensure that the lessons of the pilot phase appropriately inform the development of the NCS roll out?
The achievements of the pilot NCS projects, and the difficulties they encountered, have potentially important implications for the implementation of similar initiatives. Five aspects in particular appear to be significant: the ambitious goals of the policy; the preference for 'locally grown' solutions; the speed of implementation; the temporary nature of the funding; and the short lifetime of the pilot projects as initially planned.
How can the professional capacity and expertise developed in the pilot best feed in to future development?
The NCS pilot was an ambitious initiative that received considerable investment from SEED over three years, as well as significant additional funding from other agencies. The pilot projects were intended to modernise schools and promote social inclusion amongst children, families and communities and improve educational, health and other social outcomes. Their experiences were intended to inform the subsequent wider implementation of NCS policy. Not surprisingly, given the ambitious nature of the policy, the pilots have been unable to demonstrate substantial impacts on all these fronts. A positive feature of the pilot was SEED commitment to facilitate networking and learning amongst Phase 1 projects. SEED also developed a strategy to inform subsequent phases and roll out through the provision of materials for staff development and training, regular seminars for IMs, conferences, multi-agency staff development, an NCS Newsletter and the NCS website. These provided a range of guidance and opportunities for dialogue amongst participants. Although there have been sustained attempts at dissemination, the extent to which findings and approaches are proving transferable remains to be tested in the roll out.
In part this is because of the policy's preference for 'locally grown' solutions. One clear finding from the evaluation is the wide variety in emphases, implementation strategies and achievements across the pilot projects. Related to this issue of local autonomy has been the speed of implementation. The evaluation has shown that the 'rushed' start militated against detailed advanced planning and the extent to which key professionals were able to bring other partners and professionals fully on board from the beginning.
The effect of the speed of implementation was compounded by the temporary nature of the funding. Many projects spent the first year finding their feet and appointing staff and therefore were not able to begin implementing strategies until the second year. There were also difficulties in attracting and retaining staff. This, and the related uncertainty, limited the extent to which some projects were able to put in place exit strategies and promote sustainability. Given that the Phase 1 NCSs were attempting to bring about radical changes in areas characterised by a range of challenges, the speedy implementation and short-term funding of the pilot resulted in only limited impacts initially in some projects. The nature and level of change anticipated in the NCS policy may be most appropriately viewed as a longer-term undertaking.
Any pilot project requires a time-limit, but a period of three years may be insufficient for the Phase 1 projects to demonstrate a significant impact on attainment. There was evidence of some general improvements in pupil outcomes in line with national trends, but no clear evidence that the pilot had led to a narrowing of the attainment gap between schools participating in Phase 1 and others. However, it is not possible to establish whether the relative size of the attainment gap would have increased without the NCS initiative. Reforms take time to bed in and most NCS strategies were only indirectly focused on raising attainment levels. In view of this it is unlikely that the kinds of strategies adopted would have any clear impact on outcome data in the short term. Also, it should be noted that the statistical indicators analysed only covered two of the three years of the pilot phase and thus conclusions about impacts on pupil outcomes should be regarded as tentative. The short lifetime of the Phase 1 pilot may have limited the ambitions of some projects, eg some reported they had not pursued various difficult aims, such as those directed at community engagement, because they felt it was not possible to effect major change over such a short timescale.
In these circumstances, the achievements of the pilot NCSs should not be underestimated. Although it is difficult to isolate the 'NCS effect' from that of other associated interventions, the evaluation indicated that inter-agency liaison and practice were substantially enhanced in many projects and that multi-agency initiatives had been promoted. Professional perceptions indicated an increased capacity to meet pupil needs and improved pupil attitudes and motivation. There is little doubt that most NCS projects offered an increased range of activities and services, especially those targeted towards vulnerable groups. Many offered new school-based developments to promote pupil engagement outside the usual curriculum framework with an increased range of after-school and holiday activities. Particular developments relating to transition arrangements, breakfast clubs, study support and health education were evident. There were general indications of positive progress towards the creation of health promoting schools.
The impact of the Phase 1 pilot relied on a cocktail of approaches, linking NCS with other initiatives in health education and social work and drawing on a range of funding sources. The NCSs acted in many instances as a catalyst for change and innovation, and increased multi-agency approaches, staff development and communication were evident. The provision of additional activities undoubtedly had a positive impact on the experiences of pupils and vulnerable groups in particular. If these positive gains are to be built on and extended throughout schools participating in the NCS policy, it will be necessary to address some of the difficulties experienced by the pilot projects. A number of studies of improvement initiatives in different contexts (Barber & Dann, 1996; Stringfield, Ross & Smith, 1996) have stressed the advantages of a clear focus and a manageable set of goals. They also emphasised the need to address teaching and learning processes directly and to encourage monitoring and evaluation if significant changes in pupil outcomes are to be achieved. However, the advantages of a greater focus on ways of improving measurable pupil outcomes may need to be balanced with the broader aims of NCSs, the desire to maintain local autonomy and to stimulate local solutions to promote inclusion and combat disadvantage. The further establishing of principles of multi-agency working and service delivery in community and school life will depend on appropriate management and governance structures and realistic funding, as well as strong commitment of key individuals in local authorities from all services (education, health and social work), if NCS approaches are to become widespread nationally.
Barber, M & Dann, R (Eds) (1996) Raising Educational Standards in the Inner City: Practical initiatives in action, London: Cassell.
Scottish Office (1998) New Community Schools: The Prospectus, Edinburgh: The Scottish Office.
Stringfield, S, Ross, S & Smith, L (1996) Bold Plans for School Restructuring. The New American Schools Design, Mahwah NJ: Erlbaum.
The views expressed in this Insight are those of the author(s) and do not necessarily reflect those of the Scottish Executive or any other organisation(s) by whom the author(s) is or are employed.
Copyright © August 2003, Scottish Executive Education Department
ISSN 1478-6788 (Print)
ISSN 1478-6796 (Online)
Insight may be photocopied for use within your own institution.
A limited number of additional copies can be obtained from the Dissemination Officer, Research, Economic and Corporate Strategy Unit, Scottish Executive Education Department, Victoria Quay, Edinburgh EH6 6QQ (telephone 0131-244 0092). Copies of Insight and our other publications can also be downloaded from our website: www.scotland.gov.uk/insight
The Insight Series
1. Classroom Assistants: Key Issues from the National Evaluation
2. The Impact of ICT Initiatives in Scottish Schools
3. Moving On to Primary 1: An Exploratory Study of the Experience of Transition from Pre-School to Primary
4. Accelerating Reading and Spelling with Synthetic Phonics: A Five Year Follow Up
5. Assessment of Benefits and Costs of Out of School Care
6. Meeting the Needs of Children from Birth to Three: Research Evidence and Implications for Out-of-Home Provision
7. Key Findings from the National Evaluation of the New Community Schools Pilot Programme in Scotland
If you have views on Insight or wish to find out more about SEED's research and economics programme, please contact the Research, Economic and Corporate Strategy Unit, Scottish Executive Education Department, Victoria Quay, Edinburgh EH6 6QQ.
1 Note that for the purpose of this publication, all schools not in pilot Phase 1, 2 or 3 of the NCS Programme are defined as roll out although some may not participate until 2007.
2 The surveys adopted a rating scale approach to identify the level of particular activities or perceived impacts, as judged by respondents, with four points covering 'none/minimal', 'limited', 'moderate' or 'considerable'.
3 Data were only available for the three consecutive years 1998/99 to 2000/01 within the timeframe of the national evaluation (i.e. covering baseline to end of Year 2 of the pilot Phase 1). It is recognised that this covers only two of the three years of the pilot and thus provides a very short time scale in which to measure any change in pupil outcomes.