We are testing a new beta website for gov.scot go to new site

Evaluation of Big Noise, Sistema Scotland - Research Findings

Listen

Based on the El Sistema model in Venezuela, Sistema Scotland has established the Big Noise Children's Orchestra in Raploch in Stirling with the aim of achieving long term social transformation in an area that has faced economic and social deprivation. GEN was commissioned to evaluate the programme, to build an understanding of its impact, assess its processes and identify what elements of the programme are most important for achieving positive outcomes.

Main Findings

  • Big Noise has been successful in engaging with 80% of the 344 children who attend school or nursery in the Raploch Campus. Those who attend on a voluntary basis after school and during school holidays account for 65% of the 219 children who are eligible to attend (those in Primary 2 to Primary 7).
  • There is strong evidence that children involved in Big Noise experience benefits, primarily around personal and social development, for example improving confidence, self esteem, social skills and the ability to concentrate.
  • As a result of Big Noise, families report improved relationships at home, wider social networks and more shared activities between parents and children. It has also allowed the parents to see a more positive, aspirational future for their children and has engendered a sense of pride in what they have and will go on to achieve in their lives.
  • While the vast majority of any impact has been on the children and families who are involved in Big Noise, many of the parents and professionals interviewed during the evaluation believe that Big Noise is changing perceptions of Raploch. There is also evidence of skills exchange with partners Big Noise works with, such as the nursery and reports that the positivity that surrounds Big Noise can be beneficial for engendering positive relationships between parents and statutory services.
  • The evaluation shows that Big Noise is contributing to the achievement of eight National Outcomes; we live our lives free from crime, disorder and danger; we have strong, resilient and supportive communities where people take responsibility for their own actions and how they affect others; we are better educated, more skilled and more successful, renowned for our research and innovation; our young people are successful learners, confident individuals, effective contributors and responsible citizens; our children have the best start in life and are ready to succeed; we take pride in a strong, fair and inclusive national identity; we live longer, healthier lives; and we have tackled the significant inequalities in Scottish society. Additional longer term outcomes are expected against five of these National Outcomes, and so evidence will be gathered through longer term tracking of children and families and public data sources.
  • The processes used by Big Noise were assessed against 8 factors which have been identified by the Critical Hours: After School Programmes and Critical Success study as being associated with positive outcomes for children and were found to incorporate all of the factors. In addition, Big Noise has a number of unique features that set it apart from other interventions provided to children and families; Intensity and Immersion, Inclusion, Partnership Working and Excellence and Profile.
  • Measuring the impact of Big Noise on children's engagement with and achievement in learning has been challenging and evidence is currently limited. The evaluation found that there is potential for Big Noise to have a positive impact on children's learning, and so working more closely with schools to develop ways of measuring this impact is important.

Background

Big Noise aims to work with children from an early age to achieve long term social transformation. This type of early intervention approach is aligned with the Scottish Government's Early Years Framework. The EY Framework maintains that early intervention is key to improving the outcomes for Scotland's children, in particular, those who face disadvantage. It aims to tackle inequalities by shifting the focus of interventions from crisis management to prevention. Big Noise activities are aligned with the Curriculum for Excellence and Getting it Right for Every Child.

Big Noise is a new and ambitious programme of activities the aims of which aims are aligned with the principle policy areas associated with reducing inequalities through early intervention. As such, its impact and approach is of interest to policy makers, stakeholders and partners.

Aims and Objectives

GEN was commissioned by the Scottish Government to evaluate the Big Noise programme, drawing key lessons for delivery of programmes elsewhere. The specific objectives of the evaluation were to:

  • develop a better understanding of the impact of Big Noise;
  • examine the processes in the programme and identify opportunities for improvement;
  • examine how Big Noise operates to explore how outcomes are achieved; and
  • support the development of a self-evaluation framework.

Methodology

The evaluation team employed a mixed method approach involving desk research; interviews with Big Noise staff, volunteers and Board members; interviews with a wide range of partners; interviews with parents; and interactive workshops with children.

Outcomes

Outcomes for Children

Big Noise is having a positive impact on children's personal and social development. In particular, as a result of Big Nose, children have improved confidence, happiness, social skills, and concentration. These impacts can be even more marked for children with Special Education Needs or who are facing personal or educational challenges.

There is limited evidence at this stage of whether Big Noise is having an impact on attainment and engagement with education. There are challenges with attributing these types of outcomes to Big Noise activity. By continuing to improve partnership working with the schools, it should be possible in future to identify a way of measuring these impacts.

In the longer term there is an expectation that Big Noise may lead to reduced anti-social behaviour, improved health and wellbeing, improved employability, and greater levels of active citizenship.

Impact on Families, Communities and Organisations

Parents report that they are feeling more positive about their child's future, they themselves have made new friends, and their family does more activities together as a result of Big Noise. For those who are most involved through volunteering or the adult orchestra, the impacts can be significant. They not only meet new people, but develop confidence and skills that help them make wider changes in their lives.

At the community level many of the parents and some of the professionals who were interviewed believe that Big Noise is helping to change perceptions of Raploch for the better.

Big Noise has the potential to impact on the working practices of its partner organisations. There is evidence that the nursery staff use techniques in their teaching that they learned from the Big Noise musicians. Similarly Big Noise use techniques they learned from the nursery staff.

Senior executives at Stirling Council have reported that Big Noise is helping families to see the council as being about more than just social services and schools. They notice a shift in the balance of power, demonstrating a growing equality of relationship between families and statutory agencies, as well as a growing confidence in the families. This is likely to help Stirling Council build positive community engagement in Raploch.

Impact on National Outcomes

Big Noise is taking an early intervention approach and so contributes to ensuring that "our children have the best start in life and are ready to succeed". The orchestra activities and public performances are building children's confidence, team working skills, they are learning new skills and learning to contribute effectively to the orchestra, thereby ensuring "Our young people are successful learners, confident individuals, effective contributors, and responsible citizens". Children and adults are learning new skills and there is potential for Big Noise to impact on educational attainment, ensuring "We are better educated, more skilled and more successful, renowned for our research and innovation". The latter outcomes is one where longer term impacts can be expected in future.

By providing a safe out of school environment and working with partners to support and identify children at risk, Big Noise is contributing to "We live our lives safe from crime, disorder and danger". Children are working together in the orchestra to achieve a common goal, adults are engaging in a variety of ways which contributes to "We have strong, resilient and supportive communities where people take responsibility for their own actions and how they affect others". The holistic approach taken by Big Noise to care for the whole child, and the emphasis it places on building confidence and self-esteem contributes to "We live longer, healthier lives". Finally, Big Noise is contributing to "We have tackled significant inequalities in Scottish society" by engaging 80% of children and families, and proactively engaging with some of the most vulnerable families. For each of these 4 outcomes there are expected to be longer term impacts that will be measured over the longer term.

Processes

The processes used by Big Noise were assessed against 8 factors which have been identified by the Critical Hours: After School Programmes and Critical Success study as being associated with positive outcomes for children and were found to incorporate all of the factors. In addition, Big Noise has a number of unique features that set it apart from other interventions provided to children and families.

Intensity, Immersion and Inclusion

Intensity and Immersion

The number of hours that are delivered every week and during holidays makes Big Noise largely unique in its intensity. This provides children with a structure to their days and their weeks which may be lacking in some of their home lives. It is immersive in that it introduces children to playing in an orchestra from the start. The programme is holistic, designed to take care of every aspect of the child's needs so as well as the musical activities, they learn about healthy eating and are helped with any travel issues or barriers to attendance. Attending and participating in performances gives children and families a vital opportunity to build their confidence and aspirations.

Inclusion

Big Noise proactively aims to be inclusive through its special educational needs ( SEN) programme; its flexible approach to overcoming individual barriers to engagement, including the development of an inclusion programme for children whose challenging behaviour may have otherwise meant they were excluded from the programme; and features such as the central location of the Community Campus, the scheduling of activities and the fact that it is provided free of charge.

Partnership Working

At the strategic level, Big Noise has been able to build high profile and influential partnerships that have been instrumental in generating funding and in-kind support. Its strong strategic partnership with Stirling Council has been fundamental in enabling Big Noise to become embedded in the Community Campus and to work with children in nursery and Primary 1 during school and nursery time.

Operationally, Big Noise has built partnerships with a range of statutory agencies. Schools are its most important partner and it has aligned its activities with the Curriculum for Excellence. Big Noise has worked hard to develop these relationships, and is making good progress.

A strategic decision was taken with Stirling Council during the set-up phase to align Big Noise with the Council's Education Department, rather than with the Communities and Culture directorate as was originally planned. This meant that school staff were not as involved as they should have been in the very early stages of the implementation of Big Noise with the result that achieving good buy-in and partnership working with school staff has been problematic. Steps are being taken to address this, some progress has been made and so this partnership is expected to be more effective and productive going forward.

Excellence and Profile

The strategic partnerships that Big Noise developed were critical to developing a programme that promotes excellence, is high profile and is of national significance.

An extensive and innovative publicity campaign that included musicians performing in the street, and children and parents promoting the programme, has ensured that Big Noise has a high profile in the Raploch community.

The personal qualities that Big Noise staff bring to the role is considered fundamental to delivering a programme of excellence and also maximises inclusion. Big Noise has developed an effective recruitment, induction and staff development programme for its paid staff and volunteers.

Conclusions

Overall, there are 4 features of Big Noise that are fundamental to achieving the goal of social transformation:

  • it works with children from an early age;
  • it is totally inclusive so that it reaches and sustains the engagement of the hardest to reach families
  • it is a long term intervention, working with children from nursery to P7 and plans to work with the children through to adulthood;
  • it is delivered on a community wide scale so by achieving outcomes for individuals it will create wider spread change.

Big Noise activities are achieving short and medium term outcomes and there is overall agreement that it will achieve and sustain longer term outcomes. If it is able to continue to work with children from an early age, to provide holistic support to take care of the whole child, sustain engagement over time and engage with the majority of children in Raploch, then it has the potential to achieve more than a series of positive outcomes. It may, as part of a programme of regeneration, achieve social transformation.

This document, along with full research report of the project, and further information about social and policy research commissioned and published on behalf of the Scottish Government, can be viewed on the Internet at: http://www.scotland.gov.uk/socialresearch. If you have any further queries about social research, please contact us at socialresearch@scotland.gsi.gov.uk or on 0131-244 7560.