Evaluation of the Adaptation Scotland Programme

4. Contribution Analysis

The contribution claims have been developed based on the ToC (see chapter 3), where key assumptions have been mapped out and agreed upon by key stakeholders with strategic oversight of the Programme. This provides an overall framework for developing a credible ‘contribution story’ that links the evidence available to underlying intervention logic.

To conduct a contribution analysis, the following six steps will be taken, as set out by Mayne (2015),^[3] and adapted to the context of the Adaptation Scotland Programme:

• Setting out the attribution problem to be assessed. This step has been conducted in the development of the ToC, where the causal chains have been specified and the causal questions identified in the evaluation questions.
• Developing the ToC, desired pathways being realised, and risks. This step has already been undertaken and has been drafted (see chapter 3).
• Gathering existing evidence to test the ToC critical pathways, and underlying assumptions. This step will be conducted through the primary data collection and analysis of secondary and monitoring data (see Chapter 7). The primary and secondary data collection will be designed to gather evidence that will test whether critical pathways have materialised, and an assessment of how the Adaption Scotland Programme has contributed to this, so that the evidence can systematically be matched to the framework.
• Assemble and assess the contribution story and challenges to it. A list of contribution claims for the Adaptation Scotland Programme have been included below. At the end of the primary data collection and secondary and monitoring data collection period, these different data types will be analysed to determine the extent to which (a) change was observed, (b) the observed evidence supports or refutes the ToC’s assumptions about direction of and drivers of change, and (c) for each anticipated critical pathway, what other factors may have made an equal, lesser, greater, or alternative contribution. An evidence analysis framework will be developed to code the data from different collection methods so it can identify the strength of the evidence, whether it confirms or refutes the causal claim, and others.
• Seeking out additional evidence, where required and feasible, to strengthen the evidence base.
• Revising, and where additional evidence permits, strengthening the credibility of the contribution story. This will be under continual review during the fieldwork and analysis stages of the evaluation and will be discussed with Scottish Government if necessary.

Contribution claims

The following sub-sections present a high-level breakdown of the hypothesised contribution story, based on the ToC, along with the key indicators used to establish contribution.

Contribution claim 1 (CC1): The Adaptation Scotland Programme has contributed to the development of a growing network of leaders across Scotland who are driving effective adaptation action.

The Programme engages more businesses, communities and public bodies to form and grow networks in place-based approaches to climate adaptation. These growing networks support leaders to increase their knowledge and understanding of what adaptation actions are needed and how to act. This enables leaders to take more ownership and responsibility to drive adaptation action and share knowledge on adaptation.

CC1 relates to Adaptation Scotland Programme Outcome 1: ‘Scotland has a growing network of leaders from across society who are empowered and working together, using their influence to drive adaptation action across sectors and places.’, and ToC outcome ‘Scotland has a growing network of leaders from across society who are empowered and working together, using their influence to drive adaptation action across sectors and places’.

The Programme engages more businesses, communities and public bodies to form and grow networks in place-based approaches to climate adaptation. These growing networks support leaders to increase their knowledge and understanding of what adaptation actions are needed and how to act. This enables leaders to take more ownership and responsibility to drive adaptation action and share knowledge on adaptation.

Figure 4.1 shows the causal pathways (or ToC elements relevant to the claim). More detail on the key indicators that will be used to measure the claim and where data will be sourced from, and key risks and assumptions relating to the claim can be found in Chapter 6.

The key risks in the theory of change that were identified as relevant to this contribution claim were:

• Short-Termism and Political Will - The Programme may face challenges with short-termism, where focus is on immediate actions rather than long-term strategies. Political will and potential weariness could affect support and trickle down negatively.
• Perception of climate change impact - There is a risk if the implications and severity of climate impacts are not fully understood by organisations, businesses and people.
• Collaboration Mechanisms - Existing collaboration mechanisms might be insufficient, lacking legislative support or incentives to drive engagement and systematic change.

The key assumptions that were identified as relevant to this claim were:

• Capacity and Willingness - It is assumed that organisations and businesses are not only willing but also have the capacity to engage with adaptation initiatives, requiring both interest and resources.
• Engagement in Activities - There is an assumption that individuals and groups have the desire, time, and flexibility to engage with the Programme.
• Actionability of Advice - Communities and businesses can act upon the advice and knowledge shared by the Programme.
• Awareness and Reach of Programme - The effectiveness of the Adaptation Scotland Programme relies on the assumption that the right potential users are aware of its existence and that the Programme effectively reaches those in need of support via appropriate routes to share good practice.

Alternative explanations (or other key factors that may influence) relevant to CC1:

Third sector, community-led initiatives not using AS support/resources (e.g. advocacy/campaign groups – Stop Climate Chaos Scotland, Friends of the Earth).

Participatory processes (Scotland’s Climate Assembly, Scotland’s Children’s Parliament).

Direct experience of climate change impacts (e.g. extreme weather events).

Legislative and policy drivers (climate change act, public bodies duties).

Work of other organisations/networks (e.g. ClimateXChange, SSN, community organisations not receiving AS support, Culture for Climate Scotland).

Other knowledge sharing, public awareness and/or engagement campaigns not supported by AS (e.g. Universities, UK Centre for Greening Finance and Investment, Scottish Cities Alliance).

Contribution claim 2 (CC2): The Adaptation Scotland Programme has contributed to the development of effective approaches to support fair, just and sustainable adaptation.

The Programme engages with specialists (i.e. practitioners, researchers, institutions and policy teams) to increase understanding of what effective adaptation looks like, connecting specialists and networks to encourage cross-sector collaboration and (co-) develop tools and resources. These tools and resources are used by public sector bodies, communities and businesses to develop effective approaches for tackling adaptation challenges which support a fair, just and sustainable adaptation.

CC2 relates to Adaptation Scotland Programme Outcome 5: ‘Scotland pioneers innovative approaches to tackling adaptation challenges across finance, land and climate justice. Cross sector collaborations are developing approaches that support fair, just and sustainable adaptation’ and Outcome 2: ‘Innovative adaptation tools, resources and processes are enabling organisations to take effective adaptation action and learn from what works ‘ and ToC outcomes ‘Cross sector collaborations are developing approaches that support fair, just and sustainable adaptation’ and ‘Individuals, communities, and organisations are developing effective approaches to tackling adaptation challenges across finance, land and climate justice.’

Figure 4.2 shows the causal pathways (ToC elements relevant to the claim). More detail on the, key indicators that will be used to measure the claim and where data will be sourced from, and key risks and assumptions relating to the claim can be found in Chapter 6.

The key risks in the theory of change that were identified as relevant to this contribution claim were:

• Funding Uncertainty - Uncertainty related to funding continuity, particularly outside of direct Programme funding, poses a risk if other supportive initiatives, such as climate action hubs, lose funding.
• Collaboration Mechanisms: Existing collaboration mechanisms might be insufficient, lacking legislative support or incentives to drive engagement and systematic change.
• Inclusivity: The risk that vulnerable or key groups may be overlooked if the Programme is not sufficiently integrated into broader systems or if lines of communication fail.
• The key assumptions that were identified as relevant to this claim were:
• Actionability of advice – communities and businesses can act upon the advice and knowledge shared by the Programme.
• Awareness and Reach of Programme - The right potential users are aware of its existence and that the Programme effectively reaches those in need of support.

Alternative explanations (or other key factors that may influence) relevant to CC2:

• Other policy interventions (e.g. Just Transition Framework, public engagement and Open Government strategies, community empowerment act).
• Third sector, community-led initiatives not using AS support/resources (e.g. advocacy/campaign groups – Stop Climate Chaos Scotland, Friends of the Earth).
• Other funding streams (e.g. engagement and community funds, just transition funds, lottery).
• Direct experience of climate change impacts (e.g. extreme weather events).
• Work of other organisations/networks (e.g. ClimateXChange, SSN, community organisations not receiving AS support, Culture for Climate Scotland).
• Other knowledge sharing, public awareness and/or engagement campaigns not supported by AS (e.g. Universities, UK Centre for Greening Finance and Investment, Scottish Cities Alliance.
• The Programme cannot directly control the value of funding secured by regional partnerships.

Contribution claim 3 (CC3): The Adaptation Scotland Programme has contributed to developing place-based initiatives that drive effective adaptation action across cities, regions and localities, aligned with the interests and needs of communities.

Adaptation Scotland has embedded the Place Principle into the foundation of the Programme. The Programme has helped to establish, expand and support regional adaptation partnerships to enable decisions on place-based approaches. This has led to the design and delivery of initiatives that drive effective adaptation action based on the interests and needs of communities.

CC3 relates to Adaptation Scotland Programme Outcome 3: ‘Place-based initiatives are driving effective adaptation action across cities, regions and localities, and sharing their learning and resources, guided by the Place Principle and aligned with the interests and needs of communities. and the ToC outcome ‘Place-based initiatives are driving effective adaptation actions across cities, regions and localities, and sharing their learning and resources, aligned with the interests and needs of communities’.

Figure 4.3 shows the causal pathways (or ToC outcomes elements relevant to the claim).

More detail on the key indicators that will be used to measure the claim and where data will be sourced from, and key risks and assumptions relating to the claim can be found in Chapter 6.

The key risks in the theory of change that were identified as relevant to this contribution claim were:

• Staff changes – changes or departures in personnel in both partner and delivery organisations could disrupt progress.

The key assumptions that were identified as relevant to this claim were:

• Actionability of advice – communities and businesses can act upon the advice and knowledge shared by the Programme.
• Awareness and Reach of Programme - The right potential users are aware of its existence and that the Programme effectively reaches those in need of support.
• Engagement in activities – individuals and groups have the desire, time and flexibility to engage with the Programme.
• Support from Local Structures - For place-based initiatives to succeed, there is an underlying assumption that they can gain access to local authority and community support, enabling integration into decision-making structures.
• Perception of Climate Change Impact: Organisations, businesses and people recognise how climate change impacts their operations.

Alternative explanations (or other key factors that may influence) relevant to CC3:

• Other policy interventions (e.g. Just Transition Framework, public engagement and Open Government strategies, community empowerment act).
• Third sector, community-led initiatives not using AS support/resources (e.g. advocacy/campaign groups – Stop Climate Chaos Scotland, Friends of the Earth).
• Participatory processes (Scotland’s Climate Assembly, Scotland’s Children’s Parliament)
• Other funding streams (e.g. engagement and community funds, just transition funds, lottery).
• Direct experience of climate change impacts (e.g. extreme weather events).
• Work of other organisations/networks (e.g. ClimateXChange, SSN, community organisations not receiving AS support, Culture for Climate Scotland).
• Other knowledge sharing, public awareness and/or engagement campaigns not supported by AS (e.g. Universities, UK Centre for Greening Finance and Investment, Scottish Cities Alliance).

Contribution claim 4 (CC4): The Adaptation Scotland Programme has contributed to an increase in knowledge, engagement and capacity in shaping and delivering adaptation policy and action in Scotland.

Adaptation Scotland regularly shares the work and findings of the Programme, and insights from wider relevant adaptation work, to its stakeholders. More stakeholders are engaging with the Programme to shape and deliver adaptation policy and action in Scotland. Stakeholders have a greater capacity and feel more empowered to shape and deliver policy and action in Scotland.

CC4 relates to Adaptation Scotland Programme Outcome 4: ‘People are increasingly engaged in shaping adaptation policy and action in Scotland. The views of those most vulnerable to climate change are valued and acted upon.’, and ToC outcomes ‘People are increasingly engaged in shaping adaptation policy and action in Scotland’ and ‘Organisations, people and places are equipped with the knowledge to help deliver change’.

Figure 4.4 shows the causal pathways (or ToC elements relevant to the claim). More detail on the, key indicators that will be used to measure the claim and where data will be sourced from can be found in Chapter 6.

The key risks in the theory of change that were identified as relevant to this contribution claim were:

• Collaboration mechanisms – existing collaboration mechanisms are insufficient, lacking legislative support or incentives to drive engagement and systematic change.
• Legislative support – absence of robust legislation to incentivise and enforce private sector changes could be a barrier to achieving comprehensive adaptation outcomes.
• Staff changes – changes or departures in personnel in both partner and delivery organisations could disrupt progress.
• Measuring relevant outcomes – reliance on quantitative data means qualitative insights are missed, meaning outputs being measured do not accurately reflect the benefits or knowledge imparted.

The key assumptions that were identified as relevant to this claim were:

• Actionability of advice – communities and businesses can act upon the advice and knowledge shared by the Programme.
• Capacity and willingness – organisations and businesses are not only willing but also have the capacity to engage with adaptation initiatives, requiring both interest and resources.
• Engagement – individuals and groups have the desire, time and flexibility to engage with the Programme.

Alternative explanations (or other key factors that may influence) relevant to CC4:

• Direct experience of climate change impacts (e.g. extreme weather events).
• Legislative and policy drivers (climate change act, public bodies duties).
• Work of other organisations/networks (e.g. ClimateXChange, SSN, community organisations not receiving AS support, Culture for Climate Scotland).
• Other knowledge sharing, public awareness and/or engagement campaigns not supported by AS (e.g. Universities, UK Centre for Greening Finance and Investment, Scottish Cities Alliance).
• Other funding streams (e.g. Climate Action Hubs, Sustainable Scotland Network, Climate Intelligence Service).

Contribution claim 5 (CC5): The Adaptation Scotland Programme has contributed to bettering the value of and acting upon the views of the most vulnerable to climate change at both national and local level.

The Programme has worked with community and third sector organisations to engage groups most vulnerable to climate change and connect them to the networks. Groups most vulnerable to climate change are empowered to share their views and input to effective adaptation resources. Their views are valued and acted upon in adaptation actions at the local and national level.

CC5 relates to Adaptation Scotland Programme Outcome 4: ‘People are increasingly engaged in shaping adaptation policy and action in Scotland. The views of those most vulnerable to climate change are valued and acted upon.’, and ToC outcome ‘The views of the most vulnerable to climate change are valued and acted upon.’

Figure 4.5 shows the causal pathways (or ToC elements relevant to the claim). More detail on the key indicators that will be used to measure the claim and where data will be sourced from can be found in Chapter 6.

The key risks in the theory of change that were identified as relevant to this contribution claim were:

• Inclusivity – vulnerable or key groups may be overlooked if the Programme is not sufficiently integrated into the broader systems or if lines of communication fail.
• Measuring relevant outcomes – reliance on quantitative data means qualitative insights are missed, meaning outputs being measured do not accurately reflect the benefits or knowledge imparted.
• Staff changes - changes or departures in personnel in both partner and delivery organisations could disrupt progress.

The key assumptions that were identified as relevant to this claim were:

• Awareness and reach –the right potential users are aware of the Programme’s existence and that the Programme effectively reaches those in need of support.

Alternative explanations (or other key factors that may influence) relevant to CC5:

• Other policy interventions (e.g. Just Transition Framework, public engagement and Open Government strategies, community empowerment act).
• Third sector, community-led initiatives not using AS support/resources (e.g. advocacy/campaign groups – Stop Climate Chaos Scotland, Friends of the Earth).
• Poverty and social justice groups not using AS support/resources (e.g. Glasgow Community Climate Action Network).
• Participatory processes (Scotland’s Climate Assembly, Scotland’s Children’s Parliament).
• Other funding streams (e.g. engagement and community funds, just transition funds, lottery).