We are testing a new beta website for gov.scot go to new site

Research Forum 13 January

Scottish Third Sector Research Forum

Third Quarterly Meeting - 13 January 2009

Victoria Quay, Edinburgh

In attendance:

Geoff Pope, Third Sector Division

Steven Marwick, ESS

Louise Scott, ASD

Ruchir Shah, SCVO

Kris von Wald, ACOSVO

Chris Higgins, HIE

Louise Meikleham, OSCR

Silvy Peeters, ESS

Paul Tyrer, Third Sector Division

Erica Wimbush, NHS Health Scotland

Kathy Cameron, COSLA

Gerry Higgins, CEIS

Sandy Watson, Scottish Enterprise

Antonia Swinson, SSEC

Luska Jerdin, Public Service Delivery Division

Emma McGuire, Third Sector Division


Helen Harper, VDS

Eleanor Burt, St Andrews University


1. The intended outcome of the meeting was identified as follows:

Everyone to have a clear understanding of what an outcomes approach means for research, and to aim to build this in wherever possible.

2. Luska Jerdin gave an introduction to the outcomes approach and its value was provided, with the following key points:

  • Outcomes are about how we deliver services: the focus is on making a difference. Ultimately the outcomes approach will create better services for users.
  • This isn't a new approach - third sector and others have been using outcomes for some time.
  • Outcomes require a cross-cutting and holistic approach - ring-fencing would not work alongside them, hence the current focus on Single Outcome Agreements with local authorities. The change here is a huge cultural challenge.
  • In order to maximise its potential within the new arrangements, the third sector must consider its place within national outcomes. Local authorities are seeking to articulate their impact and the third sector is likely to be asked to do the same.

3. Four presentations were given on different experiences of evidence-based outcomes. Key points were as follows:

Silvy Peeters, Evaluation Support Scotland (ESS):

  • Outcomes and evidence equate to differences/changes and the demonstration of these.
  • Research and evidence are at the heart of the third sector's contribution to the national indicators.
  • Building the evidence base requires time, collaboration and resources - we need to broaden the way we think about research beyond commissioned, expert research into the work we can do for ourselves, e.g. self-evaluation.
  • ESS is promoting this agenda, helping third sector organisations, funders and policy makers learn how to take forward evaluation and use the resultant learning.
  • Getting policy right is also essential - we need evidence informed policy, not evidence backed policy.

Louise Scott, Local Government and Public Sector Reform Research:

  • The change inputs/processes/outputs to outcomes has expanded the measurement and reporting timescale, which no longer fits within political terms of office. Outcomes need to be seen longer term, and a myriad of activities contribute to them - not all within the influence of government.
  • The Scottish Government is using techniques such as contributions analysis to explore the contribution that an individual policy is making towards an overall desired effect, such as a national indicator.
  • It is important to unpack the underlying assumptions about what a policy's impact will be, to identify the areas where evidence is weak and focus resources there.

Ruchir Shah, Scottish Council of Voluntary Organisations (SCVO):

  • It is important not to confuse outcomes with performance assessment: the latter is about efficiency, while the former is about effectiveness.
  • We need to engage multiple stakeholders, at different levels, and modify our approach as appropriate.
  • Initiatives must be placed in their broader context - organisations are increasingly in intricate webs of partnerships.

Erica Wimbush, NHS Health Scotland:

  • The focus here is on outcome-focussed performance management, in the area of health improvement - looking at the external impact of internal performance.
  • NHS Health Scotland are moving away from discrete evaluation studies, and instead making use of frameworks and streams of information, e.g. monitoring national levels of alcohol consumption, and the price/availability of alcohol. This is a much more sophisticated and comprehensive modelling system for looking at impact.
  • The outcomes approach also provides a huge opportunity to work across sectors.

4. Following the presentations, attendees separated into pairs to discuss the following questions:

  • To reflect on what grabbed you from the four presentations.
  • What will help or hinder the role of research in relation to outcomes.
  • What can the forum do to respond to this agenda.

Following that the group came back together and shared key points in a discussion facilitated by Steven Marwick:

  • Reflections on the four presentations:
    • Various members noted a gulf between theory and practice. There needs to be a translation of theory into an intelligible framework for meaningful decisions.
    • Just because people can use the right language, doesn't necessarily mean they have the understanding that goes with it, therefore their behaviour will be unchanged. People need training and preparation to tackle this issue.
    • Research should be a communicator and enabler - we are looking at impact, not making judgements about whether the impact is good or bad.
    • Research is a tool for decision-making - but what does research look like, and how do we align it with the bigger picture?
    • Need to raise the understanding of evaluation as learning, and embed evaluation streams within working processes.
  • What will help or hinder the role of research in relation to outcomes?

o The wider SG understanding of the third sector needs to be improved - lack of understanding can be a hindrance.

    • Moving away from single studies towards a portfolio approach to research will be helpful.
    • Working as partners and contributing to shared outcomes will be helpful. Local intelligence is important.
    • Private research consultancies could be brought into the conversation, since they actually carry out a lot of the research - should therefore have an understanding of the outcomes agenda.
    • Need to consider the extent to which research can be devolved to local levels - this could be both a help and a hindrance.
  • What can the forum do to respond to this agenda?

o Raise a challenge to the academic community to build capacity, to respond to need and to make connections.

o Collaborate internally and externally - the government has a brokerage role here.

o Keep the evaluation framework proportionate to the third sector's capacity: the outcomes requirement needs to be relevant and deliverable.

o Bring stakeholders together and share research agendas - partnership working.

o The forum will need to be reconstituted in May 2009 - at this point it would be a good idea to revisit the remit.

o Forum members should be able to share ideas among friends, and use each other as sounding boards.

o Forum also has a 'watching brief' on the way things develop over time.

6. The Evidence Library website being produced and run by SCVO on behalf of the Scottish Government will be ready to go live towards the end of January. A demo version will be shared with the research forum for comment and feedback shortly.

7. Any other business:

  • The Scottish Social Enterprise Coalition has co-commissioned a survey (with the Social Enterprise Coalition in England) on the state of social enterprise.
  • Highlands and Islands Enterprise (HIE) now have a research officer, starting on 2 Feb, who is likely to represent them on the forum in future.
  • The research on governance, commissioned by ACOSVO, OSCR and the SVA Programme, will be shared at a forum event to be run by the Third Sector Division (TSD) on 26 February.
  • TSD will shortly be putting an update on its research programme online.

8. The next meeting will be held towards the end of April. Potential dates will be circulated for consideration shortly.