Scottish Rural Communities Policy Review: Stage 4 Final Report

10. Monitoring and evaluation

There are various approaches to monitoring and evaluation across the activities of Scottish Rural Network, Scottish Rural Action and Community Led Local Development delivering to a range of aims, including to document and assess the impacts of government spending, to capture the impacts of funding programmes, projects and events, and to enable continuous learning and improvement of funding programmes and initiatives generally.

Wide-ranging evidence was collected on these monitoring and evaluation approaches for this review. This included considering previous relevant evaluations in a literature review, reviewing available datasets, analysing responses to survey questions about experiences of monitoring and evaluation, discussing the topic in interviews where relevant, and two focused workshops on monitoring and evaluation in the Community Led Local Development programme. Evidence from these sources is summarised below.

10.1 Literature review

A literature review was undertaken on monitoring and evaluation for rural community development, with a particular focus on LEADER and Community Led Local Development. The learning from this is published in a separate report.^[37] The literature review emphasised the need to match methods such as theory-based, participatory, systems, or impact methods) to complex local contexts and policy goals. It found that theory-based approaches using logic models and Theories of Change prevail in policy evaluations as they lend themselves to understanding complex interventions. Impact evaluations such as Randomised Control Trials seek direct attribution and, while robust, are complex and costly so tend to be used less. Participatory and systems approaches offer valuable insights into community dynamics and learning.

The literature review suggests a need to balance the depth and rigour of monitoring and evaluation of rural community development with the administrative cost and burden. Overall, there was evidence of the benefits of flexible, plural evaluation frameworks supported by tools like the Organisation for Economic Co-operation and Development (OECD) criteria and Social Return on Investment (SROI), which balance rigour with proportionality, recognise local capacity, and prioritise learning alongside accountability.

Scottish Rural Network monitoring and evaluation

As a network delivered by the Network Support Unit within Scottish Government, standard governmental processes and procedures apply to Scottish Rural Network’s activities. There are no publicly available monitoring and evaluation documents for particular Scottish Rural Network activities. The Network’s involvement with events such as the Scottish Rural and Islands Parliament or the Scottish Rural and Island Community Pub Network (see Section 3.4 of the Review Report), are typically evaluated by collaborating funded organisations, focused on participant feedback. Individual funding awards from the Network (See Table 4 of the Review Report) will have aims, objectives, Key Performance Indicators (KPIs) and reporting requirements (see Section 3.5 of the Review Report).

Scottish Rural Action monitoring and evaluation

Monitoring and evaluation of Scottish Rural Action activities is led by the organisation’s staff and directors (as is appropriate for an independent charity). During the years of this review this work has focused on events, which were wide ranging, and included two Scottish Rural and Island Parliaments, the European Rural Parliament, and residential meetings as part of their work with young people (See Section 4.2 of the Review Report). Scottish Government provides objectives and Key Performance Indicators for its annual grant funding to Scottish Rural Action. Table 5 in the Review Report demonstrated that, since 2020, grant objectives and monitoring detail have varied to some degree each year.

Community Led Local Development monitoring and evaluation

Information on monitoring and evaluation within the Community Led Local Development programme was collected for this review via the Coordinators’ survey (17/20 responses), interviews, a dedicated focused workshop (22 attendees representing 16 Local Action Groups), and a workshop with individuals who support the Coordinators and Scottish Government to implement bespoke Social Return on Investment software.

Social Return on Investment

In 2022, a new assessment tool was introduced to provide Local Action Groups with UK-accredited software for monitoring the Social Return on Investment (SROI) of their funded projects (‘Social Value Engine’) (See Section 5.4 of the Review Report). Local Action Groups are required to use the software platform as a condition of their grant. The approach uses financial proxies to assign monetary values to non-financial outcomes, such as improved wellbeing or reduced carbon emissions, in order to create a SROI calculation of Community Led Local Development funded projects. As detailed in the Review Report (page 71), SROI was calculated to be at least £3.66 for every £1 spent on the projects assessed from 2022-2023. Following piloting, the software has been refined on an ongoing basis. Technical refinements of the assessment tool, and support provided for tool software users, led to an SROI calculation of £3.70 for every £1 spent over 585 projects funded in 2024-2025.

After three years of use, Community Led Local Development Coordinators and/or Local Action Group members expressed a wide range of perspectives on the tool in this review. A number of participants detailed positive benefits, such as highlighting the merits of the reports produced for both Local Action Groups and individual projects. Some participants and areas engaged significantly and positively with the SROI software, to the extent that they provided insights and approaches learned to future projects and across other programmes. These participants strongly expressed support for the approach.

Resource and time constraints were noted to be significantly limiting some Coordinators’ abilities to engage with the software, particularly new users. This issue was identified in the short term, and a “buddy” system was established by the software provider to support new users and help to address this challenge. There was some evidence of intersecting resource challenges across the Community Led Local Development programme being exacerbated by the requirements to use the software - in this sense, it was another administrative requirement for a programme already constrained by resource reductions. Issues of alignment between the SROI approach, short-term funding timescales (See Section 5.5 of the Review Report), and reporting requirements were also raised, with these not always aligning smoothly, especially given the time required for some to engage with the software. These resource issues were likely undermining the benefits of the SROI approach.

Some participants raised important questions about the software and future use. Participants identified areas for future refinement, such as the need to capture value for smaller projects (for example under £2,000) or in small rural communities where quantitative outputs may be small but locally significant (for example a single social housing unit on an island). Several expressed confusion about the increasingly sophisticated approach to calculating financial proxies (now incorporating AI) and proxy definitions. This points to the need for robust and transparent measurements appropriate for use in a (rural) community context, and of ongoing and inclusive user guidance and support. The ‘buddy’ system is a notably positive step in this regard, as are ongoing refinements to the sofware. We note similar feedback on the technical allocation formula for distributing Community Led Local Development funding to Local Action Group areas; both outcomes illustrate the importance of inclusive and transparent explanations when implementing sophisticated software and data approaches in community development contexts.

Wider project and programme effects, particularly arising from qualitative impacts, were noted by some participants to be missing from retrospective SROI evaluation of individual short-term projects. There was explicit concern raised by a handful of participants about a potential future link between project funding and high SROI values in the sense that such an alignment this could potentially influence the types of projects coming forward, with larger and more easily quantifiable projects more attractive, while small but locally significant projects may not score highly. These participants stressed a core tension between the grassroots and flexible nature of the fund and a more top down quantitative approach.

To address the issues raised above, it is critical that adequate time and resource is given to users to learn how to use SROI software over the longer term (pointing to the value of revenue/core funding), and that adequate inclusive support, such as through a buddy system is included. Standardising approaches in the medium term, such as providing proxies well in advance of reporting deadlines, may help to address administrative and resource challenges. It also important to support those who have widely adopted and gone on to champion SROI approaches, sharing learning and good practice.

Community Led Local Development and monitoring and evaluation beyond SROI

SROI is one means of capturing the impact of Community Led Local Development, alongside other approaches which may more appropriately capture human stories, unexpected outcomes, learning from failure and risk taking, and change over time.

The Community Led Local Development Coordinators’ survey identified that 82% were conducting other monitoring and evaluation in addition to the SROI software provided by Scottish Government. Those who were not doing so expressed a desire to but were hindered by a lack of resource.

Just over half (53%) of Community Led Local Development Coordinators responding to the survey thought all monitoring and evaluation activity since 2021 was not effective or was slightly effective, highlighting a reasonably high level of dissatisfaction. There was evidence that the short-term funding structure, increased focus on funding smaller projects through the funding tranches, and increased reporting through SROI software, were undermining a more strategic approach to capturing the impact and lessons from the programme across process and outcomes. Participants noted that community development is a long-term process, and that its wider impacts, as well as those of the individual funded projects, take place over many years. When funding is uncertain and if there are short delivery timeframes, it makes capturing these wider, longer-term impacts very difficult.

Participants in the place-based case studies stressed the importance of qualitative approaches, which demonstrate the human elements of the programme through capturing stories and case studies, and visiting projects. Ministers and Scottish Government staff experiencing projects in action and speaking personally to those involved through in-person visits was felt to be one of the most positive ways of demonstrating how funding was being spent, and the impact it was having. Building the capacity of funded projects to capture qualitative impact data was supported in several areas through holding in-person events at the start of funding rounds.

Participants noted that the fund is now supporting smaller projects than previously, and that monitoring and evaluation requirements should be ‘proportional’ and tailored accordingly to the level of fund, timescales, and the nature of the organisation receiving the fund according to size and timing of these projects. A desire was also expressed for an enhanced strategic national monitoring and evaluation approach to better evidence how local-level Community Led Local Development activities are meeting national objectives.

10.2 Recommendations

10A. Critically reflect on current evaluations and look to adopt participative and systems-based approaches in combination with theory-based approaches, learning from the flexible funding practices of other grant makers.^[38]

10B. Align the monitoring and evaluation of the elements in the review through a coherent policy approach underpinned by a Theory of Change (See Section 8.3).

10C. Develop a long-term monitoring and evaluation strategy (5, 10+ years), evidencing change in process and outcomes.

10D. Consider publishing the Theory of Change with the opening of funding rounds and/or annual funding proposals, allowing applicants to evaluate to the Theory of Change.

10E. Providing ongoing support for both SROI and qualitative outcomes from the Community Led Local Development programme will most fully demonstrate the benefits of the programme. We appreciate that decisions on how to implement such support will take into account resource availability.

10F. Support inclusive uptake of SROI software, including through ‘buddy’ systems and other means of supporting access by a wide range of users. Consider further explanations of the role and purpose of SROI so that users can better understand and engage. Share learning on good practice and effective use.

10G. Additional support for monitoring and evaluation approaches to ensure joined-up evaluation across the ecosystem could be beneficial. It may be appropriate for these to be provided by the Scottish Rural Network. Create templates, explainers and guides and share best practice on how funded projects can both audit and evidence impact.

Responsibility: Scottish Government