Scottish Rural Communities Policy Review: Reviewing approaches and tools for evaluating rural community development interventions

Evaluation approaches

Evaluation is key to developing, implementing and learning from policy and provides accountability for policy, strategy, programme, project and activity interventions. It is essential to the work of the Scottish Government in understanding what works for whom, where and why; what interventions offer value for money; and where resources could be allocated to make the most difference.^[5]

There are numerous approaches to evaluation, with different origins, epistemologies and theoretical groundings. Understanding why we are choosing a particular approach within a particular theoretical tradition clarifies the potential benefits and limitations of the approach and its associated methods and techniques. The United Nations Development Programme’s (UNDP’s) Independent Evaluation Office sets out five approaches^[6], which are summarised in Box 1.

Box 1: Evaluation approaches

Theory-based approaches are concerned with understanding how and why an intervention works. Including logic models, Realistic Evaluation and theories of change (ToC), these approaches aim to understand how and if programmes contribute to observable results interventions by examining pathways and mechanisms leading to outcomes. Sitting in a constructivist or realist epistemology, these approaches are ideal for complex interventions, support learning and causal analysis and tend to use mixed methods. They are resource intensive and rely heavily on the accuracy of the initial ToC for valid results.

Participatory approaches also known as empowerment evaluation engage stakeholders throughout the evaluation process in defining objectives, designing methods, analysing data, and shaping recommendations. Grounded in local knowledge and lived experience, they emphasise inclusion, equity, and social justice. Potentially less rigorous and more subjective than other approaches, they require careful management to balance diverse perspectives and maintain credibility.

Impact approaches assess changes directly attributable to an intervention using rigorous methods like randomised control trials and quasi-experimental designs and are based within a positivist epistemology. They include the use of counterfactuals to compare treatment and control groups, to measure the short- and long-term changes attributable to the intervention or programme. While robust, they are costly, complex, and may overlook broader or unintended impacts.

Systems approaches include approaches such as developmental evaluation and analyse complex interventions within dynamic, interconnected systems, focusing on relationships, context, and feedback loops. Systems approaches use methods like causal loop diagrams, participatory systems mapping, and soft systems modelling which handle uncertainty and complexity well but can be abstract and hard to translate into action, require considerable methodological expertise and be resource intensive.

Foresight approaches integrate futures thinking, often into formative evaluation, and tools such as driver mapping, visioning and wind tunnelling to anticipate potential future impacts and sustainability of interventions. Integrating foresight approaches can enhance the capacity of evaluations to actively predict and adapt to changes affecting the intervention’s environment and enable organisations to adjust their strategies.

In summary, the evaluation approach is a broad family (e.g. theory-based), encompassing a dedicated approach (e.g. Theory of Change) with accompanying methods (e.g. Social Return on Investment or case study).

Scottish Government, usually through its Analytical Service Divisions or Units and in line with the HM Treasury Magenta Book^[7], carries out three main types of evaluation:

• process evaluation, which sit in theory-based approaches and aim to answer the question, “What can be learned from how the intervention was delivered?” by examining an intervention’s implementation and the pathways by which the policy was delivered
• impact evaluation, which sit in impact-based approaches and aims to answer the question “What difference has an intervention made?” by examining the measurable changes caused by an intervention
• value-for-money which tend to sit in theory-based approaches and aims to answer the question, “Is this intervention a good use of resources?” by conducting cost-effectiveness analysis and cost-benefit analysis and the valuation of non-market impacts. This might include comparing the costs and benefits of the intervention against its original business case or impact assessment. A more developed economic evaluation would also compare the benefits and costs of other ways of achieving the same strategic objective