Evaluation of the Adaptation Scotland Programme

5. Evaluation questions

The scope of this evaluation is framed by the list of evaluation questions to be answered, as in the Sections below.

The evaluation questions are initially based on the set of draft research questions identified in the Invitation to Tender (ITT), developed in collaboration with policy colleagues in Scottish Government and Adaptation Scotland. These questions have been further developed through the development and refinement of the Theory of Change.

The questions are split into two groups, as identified in the Magenta Book,^[4] to understand the process by which the intervention (the Adaptation Scotland Programme) was implemented and the impact of the intervention. Relating this back to the components of the ToC, the inputs, activities and outputs tend to relate to the process evaluation, whereas outcomes and impacts are more relevant to the impact evaluation.

The process and impact evaluation questions and sub-questions can be found in the tables below.

Theme / Evaluation questions

Design of the Programme

• Who is involved in the design of the support and how are Adaptation Scotland collaborating with other groups across their work clusters to develop this support?

Implementation and delivery

• To what extent has the Adaptation Scotland Programme been implemented as intended?
• Who is involved in the delivery of the support and how are Adaptation Scotland collaborating with other groups across their work clusters to deliver this support?
• Is the Programme sufficiently resourced and supported to deliver against the Programme aims?
• What is working well and less well in the delivery of the Programme, and where could improvements be made? Does this differ between different types of stakeholders and places? Is there anything missing from the current Programme?
• To what extent does the Programme delivery model support progress towards outcomes?

Engagement with the Programme

• Who is receiving support from the Adaptation Scotland Programme? Is its intended audience being reached, and if not why is this?
• Are there any types of stakeholders or places not being sufficiently reached by the Programme? If not, why is this and how could they be better supported and reached by the Programme?
• How effectively does the Adaptation Scotland Programme communicate with and engage stakeholders to increase public awareness and engagement on climate risk and adaptation? (Cluster 5)

Governance

• How does the Programme governance work and what role does this play in supporting delivery of the Programme and its deliverables?

Supporting the clusters

• To what extent is the Programme supporting the public sector and its leadership to better plan and take action to adapt to climate change? (cluster 1)
• To what extent is the Adaptation Programme supporting capacity building in communities and organisations working with communities? (cluster 2)
• To what extent is Adaptation Scotland engaging with the private sector? Are there enablers and barriers in private sector engagement, and how can this type of engagement be improved? (cluster 3)
• To what extent is the Programme embedding knowledge exchange, innovation and research? How is this supporting adaptation outcomes at different levels and with different groups? (cluster 4)
• To what extent is the Programme supporting the development of collaboration and partnerships, including Regional Adaptation Partnerships? What works well and less well when providing this support? How could support to Regional Adaptation Partnerships be improved to achieve national coverage by 2029? (cluster 6)
• In what ways does the Adaptation Scotland Programme engage across national policy making? (cluster 7)

Effectiveness

• To what extent has the Programme supported a policy change in ambition on adaptation policy and helped mainstream adaptation across policy areas?
• How has this been done effectively over the course of the current Programme and how might the policy support provided to Scottish Government be improved in the future?

Theme / Evaluation questions

AS Outcome 1: Leadership

• To what extent has the Adaptation Scotland Programme supported the development of a growing network of leaders across Scotland who are driving effective adaptation action?

AS Outcome 2: Effectiveness

• To what extent have adaptation tools, resources and processes led by Adaptation Scotland supported more effective adaptation action across organisations in Scotland?

AS Outcome 3: Place-based initiatives

• To what extent has the Programme effectively driven place-based adaptation initiatives across cities, regions and localities in Scotland in a way that is aligned with the Place Principle?

AS Outcome 4: Engagement

• To what extent has the Programme contributed to an increase in knowledge, engagement and capacity in shaping and delivering adaptation policy and action in Scotland?
• To what extent has the Programme contributed to bettering the value of and acting upon the views of the most vulnerable to climate change at both national and local level?

AS Outcome 5: Cross sector collaborations

• To what extent has the Adaptation Scotland Programme contributed to the development and uptake of effective approaches by cross-sector collaborations to support fair, just and sustainable adaptation?