Evaluation of the Adaptation Scotland Programme

2. Evaluation approach

The evaluation approach applies theory-based methods, facilitating understanding of the elements driving change and identification of potential barriers to the production of environmental, economic, and social benefits. Typically, quantitative impact evaluations aim to establish a counterfactual scenario to explore what would have happened in the absence of such an intervention. However, these types of approaches are not suitable in the case of the Adaptation Scotland Programme due to the complex interplay of factors influencing the outcomes (including wider concepts/policies, contextual factors such as extreme weather events affecting certain regions/areas, and the place-based nature of the Programme driving how interventions are delivered and experienced).

Therefore, identifying a convincing or viable counterfactual will be challenging for this Programme, and as such Contribution Analysis is more appropriate as it can take account of other contributing and confounding factors and build a plausible, evidence-based narrative about the Adaptation Scotland Programme’s contribution. Contribution Analysis will compare the ToC against evidence about the contribution the Programme has made (or is making) to impact outcomes. This will enable an investigation of whether the activities can plausibly be believed to have led to the expected outcomes and impacts as outlined in the ToC, as well as how and why the observed effects materialised.

The methodology utilises mixed methods, incorporating both qualitative data collection and analysis of secondary data, to gather and assess evidence from the Adaptation Scotland Programme and use this analysis to answer the evaluation questions. The evaluation will comprise both process evaluation – exploring how Programme funding has been used and how it has been delivered, to facilitate understanding of the mechanisms through which outcomes have been achieved – and impact evaluation, to explore whether the Programme is achieving its intended outcomes, as well as any long-term outcomes where evidence is available. Qualitative fieldwork will facilitate in-depth evaluation across different areas within Scotland, reflecting the context-specific nature of climate adaptation.

Process and impact evaluation

A process evaluation aims to examine the implementation of activities and delivery pathways of a Programme, and to answer questions such as what worked well and what did not, how delivery could be improved, and how contextual factors influenced implementation. The impact evaluation will explore whether, and to what extent, the Programme is achieving its intended outcomes.

The process and impact evaluation questions have been designed to focus on key aspects of the Adaptation Scotland Programme (see chapters 5 and 6 for details):

For the process evaluation, this includes exploring the design, implementation, delivery and governance of the Programme.

For the impact evaluation, this includes exploring the key outcomes in relation to leadership, effectiveness, place-based initiatives, engagement and cross-sector collaboration.

To answer the process and impact evaluation questions, evidence will be gathered across secondary and primary data sources (as outlined in chapter 6). Fieldwork will be undertaken to allow an examination of the causality of the Adaptation Scotland Programme both at national level and at regional level.

Evaluation stages

The delivery of the evaluation will take place over six stages.

Stage 1 – scoping:

An inception meeting was held on 9^th September 2025 between Scottish Government and Ipsos to discuss the evaluation approach and any adjustments required. An inception report was subsequently provided to reflect discussions. Ipsos also attended the Adaptation Scotland board meeting on 30^th October 2025 to present the evaluation approach.

As part of the scoping phase, Ipsos reviewed relevant project documents shared by Verture and the Scottish Government. The review aimed to generate a deeper understanding of the Adaptation Scotland Programme, its delivery model and range of activities, and how these are expected to operate (if at all) as an overall portfolio to create change. This informed the subsequent interviews with key stakeholders and development of the Theory of Change (ToC).

The documents reviewed included:

• Verture’s (formerly Sniffer) tender for management of the current Adaptation Scotland Programme, which includes work plan for the first two years.
• Verture’s workplan for the second two years of the Programme.
• Adaptation Scotland progress reports, particularly for the last two years.
• Excel spreadsheets shared by Verture on 2025-26 activities and KPIs/outputs.
• Scottish Government’s Scottish National Adaptation Plan 2024-2029: Monitoring and Evaluation Framework (Scottish Government).
• Monitoring outcomes of the third Scottish National Adaptation Plan (Ricardo 2024).
• Adaptation Scotland Programme: 2025-27 overview (Verture).
• Supporting, extending and strengthening collaborations and partnerships at a regional scale 2023-2027 (Verture).
• Private sector leadership: proposed strategy for the Adaptation Scotland programme 2023-2027 (Verture).
• Public Sector Climate Adaptation Capability Framework (Verture).

Ipsos also carried out eight interviews with key stakeholders involved in the design and strategy of the Adaptation Scotland Programme. The scoping interviews aimed to further Ipsos’ understanding of the Programme and its activities, both at national level and across different areas of Scotland, and to discuss plans for the involvement of regional case studies in the evaluation (it later became apparent that this approach was not viable, which is explained below). Interviews lasted around 60 minutes and were conducted online via MS Teams. This included key members of the Adaptation Scotland board suggested to Ipsos by the Scottish Government. Ipsos developed and used a scoping interview discussion guide for the interviews, which was reviewed and signed-off by the Scottish Government.

Ipsos, working with the Scottish Government, utilised learnings from the document review and scoping interviews to refine the process and impact evaluation questions set out in the invitation to tender. The draft evaluation questions were then further reviewed following the development of the Theory of Change and finalised for incorporation into the evaluation framework (see chapter 6).

Stage 2 – development of the Theory of Change, contribution claims and evaluation framework:

Ipsos drafted an outline Theory of Change (ToC) based on the scoping phase findings, including review of project documents and scoping interviews. This outline formed the starting point for discussions in the subsequent workshops with key stakeholders (more details on the ToC can be found in chapter 3).

Ipsos conducted two workshops of between two and three hours with key stakeholders to develop the ToC for the Programme. Stakeholders included Adaptation Scotland Programme leads and other key stakeholders both within Scottish Government and externally (including from Verture and Sustainable Scotland Network (SSN)):

The first workshop focused on building the ToC, with Ipsos then refining the outline based on feedback after the workshop.

The second workshop focused on refining the ToC and drawing causal pathways between elements of the ToC. Stakeholder feedback was captured on virtual post-its during the workshop, with the ToC diagram itself refined by Ipsos afterwards (reflecting feedback provided in the workshop).

The workshops aimed to ensure that the ToC accurately reflects the expectations of the Programme from a strategic perspective. The experiences of those closest to delivery will be captured in the qualitative fieldwork stage beginning in January 2026. This approach ensured a clear distinction between the intended delivery/expected impact of the Programme (explored during scoping and ToC development stages) and its actual delivery/perceived impact (to be explored during the later data collection stage).

Following the ToC workshops, Ipsos developed contribution claims based on the ToC (see chapter 4). Further input into development and refinement of the contribution claims was sought from stakeholders to support the development of this evaluation framework. This involved a meeting with Scottish Government colleagues, and feedback received via email from Verture stakeholders.

Specifically, feedback was sought on the outcomes included in the ToC, the level of confidence that those outcomes will be achieved, and what evidence is available to measure progress. This aimed to build an understanding of how and in what ways the Programme is contributing to the intended outcomes and impacts, and what the counterfactual would have been (i.e. what would have happened in the absence of the Adaptation Scotland Programme).

Ipsos will present the agreed contribution claims and the plans for qualitative fieldwork at the next Adaptation Scotland board meeting in January 2026. The meeting will be an opportunity to discuss who to invite to take part in subsequent fieldwork.

Stages 3 and 4 – process and impact evaluation:

The process and impact evaluation fieldwork will comprise of secondary data analysis and primary data collection.

During the process and impact evaluation stage, Ipsos anticipates that the principal focus for the analysis of secondary data will be the quarterly and year-end Adaptation Scotland Programme reports. Further relevant secondary documentation may be identified for review by Scottish Government, Verture or Ipsos through discussions with stakeholders. Ipsos will agree criteria for the review of these reports with Scottish Government (see chapter).

Ipsos will develop a structured template for extracting key data and insights against each process and impact evaluation question to feed into the contribution analysis. Data will be extracted from these sources manually by the research team into a thematic framework matrix in Excel. The coded data will be triangulated and synthesised along with data from other strands to answer the evaluation questions and assess the contribution claims.

Ipsos originally planned to carry out qualitative fieldwork in five case study areas, involving:

• Interviews with public sector stakeholders (3-5 per case study, 15-20 interviews in total)
• Interviews with private sector/industry stakeholders (3-5 per case study, 15-20 interviews in total)
• Interviews with communities/ voluntary sector stakeholders (3-5 per case study, 15-20 interviews in total)

Following examination of available sample, it became apparent that this approach was not viable. It was therefore agreed with Scottish Government to interview individuals from public sector, third sector and private sector organisations representing a spread of cluster-based activity, and geographics areas within Scotland. This is explained further in Chapter 7.

Stage 5 – synthesis and analysis of data into outputs:

The evaluation will involve analysing large volumes of qualitative data from a total of 78 interviews. Ipsos’ approach to analysis of qualitative data is thematic, intended to produce findings that are both relevant to the research questions and clearly grounded in participants’ accounts of their experiences.

There will be three main stages to data analysis:

• Analysis of qualitative and secondary data. The first stage of analysis will be to collate and analyse each of the data sources individually using a thematic framework matrix in Excel. This will identify themes or patterns within data sources.
• Data triangulation. The second stage will identify findings, themes and patterns across multiple data sources. It will answer evaluation questions about specific processes and outcomes and assess whether evidence supports or challenges the ToC.
• Synthesis. The third stage of analysis will synthesise and aggregate the findings of stage 1 and 2 to answer the overarching evaluation questions and test the Programme hypotheses.

The evaluation will deliver the following outputs:

• Deliverable 1: inception report detailing each stage of the research work, and minutes of the inception meeting appended.
• Deliverable 2: ToC and evaluation framework to structure the design and delivery of the evaluation.
• Deliverable 3: emerging findings report (c. 15-20 pages) to inform ongoing Programme delivery, and future development. An outline structure will be drafted and submitted for Scottish Government review at an early stage to ensure expectations are aligned.
• Deliverable 4: qualitative fieldwork involving in-depth interviews with public sector, private sector and third sector stakeholders receiving support from the Programme, some who have not received support from the Programme, as well as stakeholders involved in Programme governance/delivery (see chapter 7 for more detail).
• Deliverable 5: final evaluation report (c. 60 pages) with an executive summary and full, clearly articulated findings relating to implementation and perceived impact, and recommendations for the next Programme. An outline structure will be drafted and submitted for Scottish Government review at an early stage to ensure expectations are aligned. Ipsos will work closely with the Scottish Government and the Adaptation Scotland Board to craft pragmatic, evidence-based recommendations.
• Deliverable 6: an accessible high-level summary findings report highlighting key findings and lessons learned.
• Deliverable 7: updated KPIs for Scottish Government to consider in a future Adaptation Scotland Programme. The updated KPIs will be clearly linked to the outcomes in the ToC and evaluation framework. Ipsos will work closely with Scottish Government and the Adaptation Scotland Board to develop these.
• Deliverable 8: in-person presentation of findings for Scottish Government and Adaptation Scotland Board members.