Scottish Rural Communities Policy Review: Reviewing approaches and tools for evaluating rural community development interventions

Reviewing evaluation approaches and methods in relevant contexts

This section reviews evaluation approaches and methods used in different contexts: community development programmes, unrestricted grant funding to community organisations, OECD evaluation principles and LEADER evaluations.

Community development evaluation approaches

ESRI Ireland: appraising evaluation approaches for community development programmes

Ireland’s Economic and Social Research Institute (ESRI) reviewed metrics and methodologies for evaluating community development interventions in relation to their usefulness to Ireland’s largest community development initiative, with a spend of approximately €36 million, the Social Inclusion and Community Activation Programme.^[8] Four conceptual approaches for an overall framework were reviewed. All approaches were theory based; none were impact based nor wholly participative, though some frameworks include some participatory methods. The approaches were:

• a logic model
• NPC’s guidance on using a Theory of Change to develop a measurement and evaluation framework^[9]
• Scottish Community Development Centre’s Achieving Better Community Development (ABCD) framework
• the Learning, Evaluation and Planning (LEAP) manual^[10]

Each approach was found to incorporate a clear statement on programme objectives, which is then linked explicitly to subsequent inputs, and to identify key outcome variables that the policy should be influencing. The research found no evidence of any systematic attempts to practically measure a counterfactual estimate of community-level outcomes due primarily to challenges of demonstrating a causal link between a policy intervention and changes in broad measures of community well-being.

Welsh Government: Community-led action for climate change

A government-funded programme study explored how community-led action on climate change could contribute directly to the (at the time) Welsh Government’s strategic objective of reducing carbon emissions by 3% per-annum. A professional consultancy developed a logic model and planned to support six community-based facilitators employed through the programme to develop a theory of change for the 18 projects the facilitators were supporting.

A study of the experiences of the community development facilitators in this evaluation process^[11] found that developing pre-designed theory of change or logic model approaches to the evaluation of community projects was problematic as the measures were neither appropriate nor feasible for capturing the outcomes of community-led environmental projects. The study reported that it is often difficult for such groups to see how output-based evaluation processes will lead to improvements within their practice; a lack of incentive for collecting data is a typical barrier to research processes within community organisations, and that community project participants experienced research fatigue and many groups had little interest in evaluating their projects through quantitative approaches or cost-benefit analysis.

Facilitators felt there should be space to adopt more participatory approaches to evaluation, which would involve the community groups in a reflexive process and provide a pathway towards meaning. This appreciation resulted in an informal alteration in methodology away from the logic model-based approach, originally proposed by the external consultants, towards a learning history approach. In turn, this pointed towards a need to reframe project failures as inevitable and ultimately beneficial, enabling understanding of interdependencies.

This study suggested a need for funders to move beyond reductionist approaches that ensure project delivery, to evaluations that recognise the value of learning and experimenting above isolated impacts; and cautioned against determining outcomes without community involvement. It suggests that Developmental Evaluation, which adapts to the “disorderly and uncertain realities of complexity rather than seeking to impose order and certainty,” offers a potential means for future community engagement.

Unrestricted grant funding to community organisations

Independent funders and grant-makers have more flexibility than governments in how and where they invest and how they capture the impact of their investment programmes than governments do. Some major UK funders are increasingly adopting an unrestricted approach to their funding and adapting the way they capture impact accordingly.

The Institute for Voluntary Action Research (IVAR) is at the forefront of research on unrestricted funding. It has found that unrestricted funding enables funded organisations to have greater control over their own spending. Flexible funding involves funders adopting more open and trusting practices that make life easier for those they fund for example by accepting risk and being realistic about how much assurance applicants can provide; being open and flexible, and proportionality with reporting requirements.^[12]

Funders are naturally concerned about capturing the impact of unrestricted grant-making: restricted grants allowing funders to claim a causal link between their grants and specific outcomes, which they use to hold charities accountable and evidence their impact. The cost of this, charities argue, is that restricted project funding hinders their work, distorts accountability, and weakens them.^[13] Research supports this view, showing the harms of restricted funding models and the unrealised potential of unrestricted funding to increase social impact.^[14] Unrestricted funding enables charities to better use resources, be more forward-looking, and achieve better outcomes in a complex environment.

A recent paper outlines a provisional framework for assessing the impact of unrestricted funding, summarised in Box 2.^[15]

Box 2: Institute for Voluntary Action Research’s provisional framework for assessing the impact of unrestricted funding

1. Organisational development: Are we helping the organisations we support to get stronger, and how can we improve our contribution?

2. Outcomes oriented: Is our funding helping organisations to deliver positive outcomes for their communities?

3. Systemic change: Are we seeing progress being made towards our long-term change goals, and where can we exert positive influence?

4. Funder performance: Does the way we fund grantees support them to deliver their best work, or could we do better?

5. Adaptation oriented: Do we understand the changing context for funded charities, and are we adapting quickly enough to better support them?

IVAR’s (a self–funding research centre at Aston Business School - formerly the Centre for Voluntary Action Research) research found that funders’ adopting unrestricted funding approaches:

• reduced the reporting burden on funded organisations
• placed a greater focus on conversations
• developed trust
• rethought timeframes away from annual cycles
• looked for objectivity and rigour by using impact frameworks and standardised interview questions
• used qualitative data coding software to add further analytical rigour and using external evaluators

IVAR suggest that funders use strategy to drive impact assessment practice, adopt a strategic learning approach, champion impact as a collective achievement, reframe outcomes as valuable sources of intelligence in supporting sustained change, and entrench reflective practice to understanding, monitoring and assessing impact.

Organisation for Economic Co-operation and Development (OECD) Evaluation Criteria

Over 30 years ago, in 1991, the OECD established common definitions for six evaluation criteria – relevance, coherence, effectiveness, efficiency, impact and sustainability – to support consistent, high-quality evaluation in international development cooperation. Updated and adapted in 2018/19, a new set of definitions and two principles to guide their use were added: applying criteria thoughtfully and adapting the criteria to the evaluations’ purpose. Box 3 illustrates how the six criteria can be considered for a specific evaluation.

Box 3: OECD evaluation criteria consideration process

When undertaking an evaluation, all six criteria should be considered inititally. However, evaluations must carefully select which criteria to focus upon, dependent upon several factors including the objectives and limitations of the evaluation and the context of the intervention.

The criteria are:

1. Relevance

2. Coherence

3. Effectiveness

4. Efficiency

5. Impact

6. Sustainability

Source: Applying Evaluation Criteria Thoughtfully | OECD

These criteria^[16] provide a normative framework used to determine the merit or worth of an intervention (policy, strategy, programme, project or activity) and serve as the basis upon which evaluative judgements are made. They help evaluators to think more deeply about the nature of an intervention, its implementation process and results in the round – but in themselves are not an evaluation framework, methodology or approach as such. Instead, they provide prompts for asking the right question during the evaluation of an intervention and ‘the logic, credibility and interpretation of evidence should be clear and follow the theory of change or logic of the intervention.’^[17]

The relevance criterion is especially useful for thinking about the dynamics of evaluating community development initiatives. It includes determining the objectives of the evaluation, the interventions’ stakeholders and beneficiaries, whom should be considered throughout (specifically ‘reaching the furthest behind first’) and potential tensions or trade-offs. It includes design considerations – namely how stakeholders’ priorities are articulated in the interventions’ objectives, its underlying theory of change and theory of action.

In applying criteria thoughtfully (principle 1), questions about roles and power dynamics are especially relevant to the local contexts for national programme evaluations. Questions such as: Who are the stakeholders, what are their respective needs and interests? What are the power dynamics between them? Who needs to be involved in deciding which criteria to apply and how to understand them in the local context? This could include questions about ownership and who decides what is evaluated and prioritised.

Norway applied the criteria in evaluating its civil society grant for developing countries which aimed to contribute to a stronger civil society with the ability and capacity to promote democratisation, improve human rights and reduce poverty. The evaluation used the criteria with some adaptations and additions. It adapted the ‘effectiveness’ criterion to define it in relation to the specific objectives of the partnership which included capacity building and added a criterion – ‘value added’ – defined as professional, organisations, financial and networking competence.^[18]

LEADER evaluations: Scotland and England

There are perhaps hundreds of LEADER evaluations from across Europe since the original programme began in 1991. Many have taken a theory-based approach, developed a Theory of Change (ToC) and assessed it using mixed methods. Several have used a Social Return on Investment (SROI) approach.

Social Return on Investment is a methodology used to measure the amount of value – social, economic and environmental - created by an intervention by applying proxy financial values. It is an aggregation of the economic value of the costs and benefits that arise from an intervention, which is expressed in a return on investment ratio, usually, “For every £1 spent on the intervention, outcomes worth £xx are generated.”

The SROI approach is guided by eight principles^[19] including stakeholder engagement, transparency, not over-claiming and verifying results. It differs to cost benefit analysis, which also applies a monetary value to outcomes, in its fundamental focus on the theory of change and stakeholder engagement. Some challenges associated with SROI include:

• data collection, and acquiring good quality outcomes data
• assumptions, including how to value outcomes such as volunteer time or increases in confidence
• attribution, particularly when applied to complex place- and system-based interventions such as LEADER

Applying the SROI principles, including not over-claiming, transparency, and verification are key in addressing these methodological and practical challenges and generating confidence in the results.

Evaluation of LEADER using SROI and cost-benefit analysis

EU level: Evaluation of the costs and benefits of the implementation of LEADER (EU-wide)

This evaluation assessed the added value of LEADER and the extent to which the increased costs of implementing the LEADER approach are justified by its additional benefits.^[20] Additional benefits – or value added – comprise improved governance, social capital and enhanced results or impacts:

• Improved governance comprises the institutions, processes and mechanisms through which public, economic and civil society stakeholders articulate their interests, exercise their legal rights, meet their obligations, and mediate their differences to manage public affairs at all levels in a collaborative manner
• Improved social capital, which is understood as a multidimensional concept, including features of social organisations such as networks, norms, and social trust that facilitate coordination and cooperation for mutual benefit
• Enhanced results and impacts of projects in terms of increased leverage, more sustainable projects, more innovative projects and new project promoters, as they compare to implementation without the LEADER method

These additional benefits are due to the application of the seven LEADER principles, and the enabling processes of the Local Development Strategy, the CAP Strategic Plan (CSP) and Local Action Group (LAG) delivery mechanism and animation/capacity building support.^[21]

The evaluation found that LEADER did generate considerable additional benefits across all three areas of ‘value added’. Of interest for this paper are the reported main limitations of data and analysis. Data gaps identified included the availability of adequate data to assess the LEADER added value elements since the monitoring requirements for the 2014-2022 programming period did not include systematic collection of information pertaining to local governance or social capital in the Local Action Groups. This was addressed through conducting a questionnaire-based survey, and to mitigate the problem of missing relevant cost information, the analysis used combined data from various sources. Despite the requirements, not all Member States/regions the detail of running costs as separated from animation costs – and not all case study Local Action Groups were able to provide separate cost figures for running and for animation.

Another challenge was highlighted regarding the comparability of LEADER and non-LEADER projects: limiting factors included the possibility of comparing LEADER projects with analogous non-LEADER projects in terms of costs and benefits experienced.

National Programme level: Evaluating LEADER in England using SROI

The Countryside and Community Research Institute in England evaluated Axis 1 (competitiveness) and Axis 3 (economic diversification and quality of life) of the 2007–13 Rural Development Programme for England using SROI.^[22] LEADER Local Action Groups had a key role in delivering funding under both Axes, in particular Axis 3. The research found that every £1 invested in Axis 3 generated a return of £2.16 for impacts already identified.

A large proportion of the benefits were reported in: increased micro-enterprises; increased opportunities for employment; increased cross-community development and regeneration through integrated village initiatives; improved service provision in rural areas and social capital, and improved community tied and strengthened civic engagement through greater use of community buildings and public spaces.

The evaluation identified a challenge with using SROI at national programme rather than individual project level. It found that whilst overall, individual data quality was good, only 10-15 beneficiaries were interviewed within each case study area (four counties made up the case study area), representing a broad range of organisations and project types, meaning that the pool of data with which to construct robust indicator values (representing average performance) is too limited at the regional level. Creating SROI ratios for individual regions was therefore not possible on the basis of only one county per region and considering the size of the beneficiary sample; therefore the data had to be analysed in aggregate to produce national estimates. This example highlights both the value and the challenges of using SROI in national programme evaluation of community development interventions.

Project level SROI: Using Social Value Engine to evaluate impact in Scottish Local Action Groups

Social Value Engine^[23] is a platform developed by Rose Regeneration and East Riding of Yorkshire Council for assessing SROI which is being used by a number of Local Action Groups in Scotland to assess the impact of CLLD funding. The Social Value Engine (SVE) uses over 700 peer-reviewed financial proxies to assign monetary values to non-financial outcomes, such as improved wellbeing or reduced carbon emissions, enabling a robust SROI calculation. The SVE aligns outcomes with local priorities and global frameworks like the UN Sustainable Development Goals The SVE platform helps project evaluation by assisting with defining project scope and objectives, selecting inputs and activities, choosing appropriate outcomes and proxies, and inputting data. Evidence for the Review on SROI is provided in the main report.