Scottish Rural Communities Policy Review: Reviewing approaches and tools for evaluating rural community development interventions

Conclusions and reflections

Theory based evaluation approaches tend to be the most common and valuable for evaluating community development initiatives. The comprehensive study by Economic and Social Research Institute in Ireland found no evidence of impact evaluations in the literature for community development programme evaluations due to the challenge of demonstrating causality between the policy intervention and community wellbeing.

Participative and systems-based approaches hold useful insight in terms of evaluating rural (bottom-up) development interventions particularly in combination with theory-based approaches.

Participative approaches, for example, stress the importance of including beneficiaries in the design of evaluation objectives and the Theory of Change – a point which acknowledges that local volunteers are critical to local community development activities^[24] – including the evaluation of them. This leads in turn to the need to create appropriate conditions for the involvement of communities in evaluation which recognises power relations and the inherent tension between top-down (i.e. government/programme led) and bottom-up approaches.

Systems approaches, particularly developmental evaluation, can help here by underscoring the importance of being open and flexible rather than rigidly wedded to an evaluation framework by integrating reflexivity and learning, especially learning from failure and unanticipated outcomes which may far exceed the expectations set in the original programme

The bourgeoning field of flexible funding by grant makers presents a warning of the innovation and impact that may be lost by setting overly strict parameters around funding and evaluation requirements – including by over-burdening and turning off local actors and groups. This speaks to the challenge of making evaluation requirements proportional to the funding and the capacity of local actors to do evaluation, particularly when their activities may be funded by multiple funding sources, each requiring different evaluations.

The OECD evaluation criteria and principles offer a useful tool for reflective evaluation practice, inviting reflexivity and a holistic approach. This tool can help with taking a step back from the immediacy of the programme delivery and evaluation, encouraging evaluation to be done some time after the programme may have finished.

SROI approaches help make an important case for the value – social, environmental and economic – of rural community development interventions. SROI is predicated on good quality data – and the assumptions and attribution which follow from it – which is not always available, especially at local level. If such good quality data is developed, there are resource implications. This reiterates the point from the SEFARI (the Scottish Environment, Food and Agriculture Research Institutes) project that local level data is often limited or unavailable at small enough scale and in joined up ways to highlight connected rural challenges and opportunities.^[25] Using proxies, as Social Value Engine does, is a potential workaround – though not without its own challenges.

In conclusion, this concept paper highlights the importance of plurality, flexibility and reflexivity in evaluating rural community development initiatives.