Evaluation of the Adaptation Scotland Programme

Findings from a process and impact evaluation of the Scottish Government's Adaptation Scotland Programme. The Programme supports the delivery of Scotland’s National Adaptation Plan.


6. Evaluation framework

This section presents the proposed evaluation framework. The tables below include the evaluation questions and the proposed indicators to help answer them. The tables also indicate whether qualitative (through in-depth interviews) and/or monitoring or secondary data will be collected and analysed against the indicator. Themes to be covered in the interviews have been prioritised for those most relevant to each audience, recognising that it will not be possible to cover each evaluation question within 60 minutes.

Table 6.1 provides the evaluation framework for the process evaluation questions, and table 6.2 provides the evaluation framework for the impact evaluation questions.

Table 6.1 – Process evaluation framework

Theme

Evaluation question

Potential indicators

Evidence type needed: Stakeholder interviews

Evidence type needed: Programme governance/ delivery interviews

Evidence type needed: Monitoring data

Evidence type needed: Secondary data

Design of the Programme

Who is involved in the design of the support and how are Adaptation Scotland collaborating with other groups across their work clusters to develop this support?

Number and profile (by sector, size, geography) of partners involved in Programme design.Volume and type of design support outputs.Partner satisfaction with collaboration.Barriers and enablers reported (e.g., capacity, data‑sharing, clarity of offer).

N/A

P

P

N/A

Implementation and delivery

To what extent has the Adaptation Scotland Programme been implemented as intended?

Planned vs delivered activities/outputs per cluster (inc. sectors and geographic coverage).Milestones achieved vs timeline.

N/A

P

P

N/A

Implementation and delivery

Who is involved in the delivery of the support and how is Adaptation Scotland collaborating with other groups across their work clusters to deliver this support?

Number and profile of partners involved in Programme delivery.Volume and type of delivery support outputs (e.g. joint trainings/events, co-authored tools/resources, joint bids/projects).Partner satisfaction with collaboration (e.g. Paul Watkiss Associates).Barriers and enablers reported (e.g., capacity, data‑sharing, clarity of offer).

N/A

P

P

N/A

Implementation and delivery

Is the Programme sufficiently resourced and supported to deliver against the Programme aims?

Total FTE staff and vacancy rates.Staff skills coverage against aims.Staff training undertaken.Availability and capacity of delivery partners.Stakeholder perceptions of timeliness, availability and continuity of support.Response times/waiting lists.

P

P

P

N/A

Implementation and delivery

What is working well and less well in the delivery of the Programme and where could improvements be made? Does this differ between different types of stakeholders and places? Is there anything missing from the current Programme?

Stakeholder satisfaction.Evidence of early/ intermediate outcomes.Planned vs delivered activities/outputs per cluster (inc. sectors and geographic coverage).

P

P

P

N/A

Implementation and delivery

To what extent does the Programme delivery model support progress towards outcomes?

Stakeholder journeys from increased awareness through to action (e.g. PSCAN engagement translating into organisational change).Number and profile of organisations adopting adaptation tools/integrating adaptation actions into their plans.Number of regional partnerships formed/strengthened.Evidence of cross sector collaboration.

P

P

P

P

Engagement with the Programme

Who is receiving support from the Adaptation Scotland Programme? Is its intended audience being reached, and if not why is this?

Documented target groups.Number and profile of organisations supported.Number and profile of requests for support which Adaptation Scotland have not been able to take upType and dosage of support (i.e. light touch or intensive).

N/A

P

P

N/A

Engagement with the Programme

Are there any types of groups or places not being sufficiently reached by the Programme? If not, why is this and how could they be better supported and reached by the Programme?

Documented target groups.Number and profile of organisations supported.Type and dosage of support (i.e. light touch or intensive).Stakeholder feedback.Advertising/engagement campaigns.

N/A

P

P

N/A

Engagement with the Programme

How effectively does the Adaptation Scotland Programme communicate with and engage stakeholders to increase public awareness and engagement on climate risk and adaptation? (Cluster 5)

Stakeholder communications.Reach of stakeholder communication (e.g. mailing list numbers, open and click through rates, event registration/attendance figures, resource downloads, social media engagement).Conversion from awareness to engagement (e.g. enquiry to attendance).Stakeholder feedback.

P

N/A

P

N/A

Governance

How does the Programme governance work and what role does this play in supporting delivery of the Programme and its deliverables?

Governance documents (e.g. organisational charts, terms of reference, risk registers, KPIs).Board meetings (frequency and attendance).Stakeholder feedback on satisfaction with governance.

N/A

P

P

N/A

Supporting the clusters

To what extent is the Programme supporting the public sector and its leadership to better plan and take action to adapt to climate change? (cluster 1)

Number and profile of public sector stakeholders engaged.Number of public bodies supported to complete/update climate risk assessments and adaptation plans/roadmaps.Integration of adaptation into corporate plans, service plans, capital Programme gateways, procurement/asset management standards.Use/adoption of AS tools/resources.New or enhanced cross-sector collaborations.

P

N/A

P

P

Supporting the clusters

To what extent is the Adaptation Programme supporting capacity building in communities and organisations working with communities? (cluster 2)

Number and profile of organisations engaged.Type and depth of engagement.Stakeholder feedback.Use/adoption of AS tools/resources

P

N/A

P

P

Supporting the clusters

To what extent is Adaptation Scotland engaging with the private sector? Are there enablers and barriers in private sector engagement, and how can this type of engagement be improved? (cluster 3)

Number and profile of organisations engaged.Type and depth of engagement.Stakeholder feedback.Use/adoption of AS tools/resources.

P

N/A

P

P

Supporting the clusters

To what extent is the Programme embedding knowledge exchange, innovation and research? How is this supporting adaptation outcomes at different levels and with different groups? (cluster 4)

Existence of mechanisms for knowledge exchange (e.g. meetings, events, training).Reach and use of knowledge exchange mechanisms.Development of research reports, tools, guidance, training.Partnerships with research bodies.Stakeholder feedback.

P

N/A

P

P

Supporting the clusters

To what extent is the Programme supporting the development of collaboration and partnerships, including Regional Adaptation Partnerships? (cluster 6)

Number of collaborations/partnerships supported.Partnership maturity.Size and diversity of partnerships.

P

N/A

P

P

Supporting the clusters

To what extent is the Programme supporting the development of these types of partnerships and what works well and less well when providing this support? How could support to Regional Adaptation Partnerships be improved to achieve national coverage by 2029? (cluster 6).

Number of collaborations/partnerships supported.Partnership maturity.Size and diversity of partnerships.Stakeholder feedback.

P

N/A

P

P

Supporting the clusters

In what ways does the Adaptation Scotland Programme engage across national policy making? (cluster 7)

AS representation on national boards, working groups.Submissions to consultations, calls for evidence.Input into national guidance/standards.Citations of AS resources in national policy documents.

N/A

P

P

P

Effectiveness

To what extent has the Programme supported a policy change in ambition on adaptation policy and helped mainstream adaptation across policy areas?

Perceived visibility of adaptation action across government policy areas.Citations of AS resources in national policy documents.

N/A

P

P

P

Efficiency and effectiveness

How has this been done effectively over the course of the current Programme and how might the policy support provided to Scottish Government be improved in the future?

Stakeholder feedback.

N/A

P

N/A

N/A

Table 6.2 – Impact evaluation framework

Contribution claim

Relevant evaluation question(s)

Key indicators (ToC outputs)

Potential additional indicators

Evidence type needed: Stakeholder interviews

Evidence type needed: Programme governance/ delivery interviews

Evidence type needed: Monitoring data

Evidence type needed: Secondary data

CC1: The Adaptation Scotland Programme has contributed to the development of a growing network of leaders across Scotland who are driving effective adaptation action.

To what extent has the Adaptation Scotland Programme supported the development of a growing network of leaders across Scotland who are driving effective adaptation action?

Cluster 1:Number and reach of engagement events with public bodies supported.Number of events targeted at senior decision makers within public bodies.Targeted promotion of resources for lower maturity sectors, including through existing sectoral events.Number and reach of resources developed for/with the public sector.Number of public bodies participating in public sector networks.Cluster 2:Number and reach of community training and learning events.Number and reach of resources developed or enhanced for/with communities or third sector.Number and reach of events for communities or third sector.Number of Climate Action Hubs supported per year.Number of engagement activities relating to Scottish Government plans for public engagement.Cluster 3:Private sector leaderships network options scoped.Co-development of approach with Enterprise Agencies.Co-ordination of final network approach, supporting Enterprise Agencies (and any other core partners identified through scoping).Number and reach of events and training sessions for enterprises/businesses.Number of tools and resources developed for/with the private sector.Number of new climate ready business tools and resources launched.

  • Documented criteria for leadership.
  • Number and profile of ‘leaders’ engaged in the Programme.
  • Self-reported increase in knowledge/skills/confidence on effective adaptation action.
  • Evidence of cross sector connections.
  • Use of adaptation tools/resources.
  • Evidence of adaptation actions taken.
  • Stakeholder feedback.

P

P

P

P

CC2: The Adaptation Scotland Programme has contributed to the development of effective approaches to support fair, just and sustainable adaptation

To what extent have adaptation tools, resources and processes led by Adaptation Scotland supported more effective adaptation action across organisations in Scotland?To what extent has the Adaptation Scotland Programme contributed to the development and uptake of effective approaches by cross-sector collaborations to support fair, just and sustainable adaptation?

Cluster 1Targeted promotion of resources for lower maturity sectors.Number and reach of resources developed for/with the public sector.Cluster 2Number and reach of resources developed or enhanced for/with communities or third sector.Cluster 3Number and reach of resources developed for/with the private sector.Cluster 4Number of items from Adaptation Scotland projects submitted as part of collaborative projects.Number of evidence calls and engagement events promoted across Adaptation Scotland networks and projects.Number of times publicly funded institutions with relevant expertise have presented at the Public Sector Climate Adaptation Network (PSCAN).

  • Use of adaptation tools and resources across sectors.
  • User satisfaction with tools and resources.
  • Increase in uptake of effective adaptation action.
  • Number of specialists engaged with.
  • Evidence of co-development of tools with specialists and networks.
  • Number and profile of AS supported cross sector collaborations.
  • Understanding and use of effective adaptation approaches.
  • Evidence of consideration of just transition principles in adaptation approaches.

P

N/A

P

P

CC3: The Adaptation Scotland Programme has contributed to developing place-based initiatives that drive effective adaptation action across cities, regions and localities, aligned with the interests and needs of communities

To what extent has the Programme effectively driven place-based adaptation initiatives across cities, regions and localities in Scotland in a way that is aligned with the Place Principle?

Cluster 6:Number of regional adaptation partnerships supported.Progression of partnerships through the red, amber and green stages of the routemap.Number of tools and resources developed for/with regional adaptation partnerships.

  • Number and profile of AS supported place-based adaptation initiatives.
  • Evidence of community participation in adaptation initiatives.
  • Evidence of cross sector connections.
  • Stakeholder feedback.
  • Self-reported gains in knowledge/skills/ confidence among participants.
  • Awareness and use of the Place Principle.

P

P

P

P

CC4: The Adaptation Scotland Programme has contributed to an increase in knowledge, engagement and capacity in shaping and delivering adaptation policy and action in Scotland.

To what extent has the Programme contributed to an increase in knowledge, engagement and capacity in shaping and delivering adaptation policy and action in Scotland?

Cluster 4Number of items from Adaptation Scotland projects submitted as part of collaborative projects.Number of evidence calls and engagement events promoted across Adaptation Scotland networks and projects.Cluster 5Number of collaborative partners actively engaged with.Communication and engagement strategy submitted to Scottish Government.Implementation of communication and engagement strategy.Optimisation of Adaptation Scotland Programme website.Publication of quarterly newsletter.Quarterly publication of impact case studies.Cluster 7Supporting Scotland's Climate Change Adaptation Plan (3), including through stakeholder mapping, workshop design and facilitation, briefing papers, evidence and case studies.Number of deliverables provided to Scottish Government policy teams and ministers.Number of interviews, case study recommendations and contacts for statutory adaptation Programme annual progress reports.

  • Self-reported gains in knowledge/skills/confidence among participants.
  • Stakeholder participation in policy processes.
  • Existence of mechanisms for knowledge exchange (e.g. meetings, events, training).
  • Reach and use of knowledge exchange mechanisms.
  • Stakeholder feedback.

P

P

P

P

CC5: The Adaptation Scotland Programme has contributed to bettering the value of and acting upon the views of the most vulnerable to climate change at both national and local level.

To what extent has the Programme contributed to bettering the value of and acting upon the views of the most vulnerable to climate change at both national and local level?

Cluster 2Number and reach of community training and learning events.Number and reach of resources developed or enhanced for/with communities or third sector.Number and reach of events for communities or third sector.Number of engagement activities relating to Scottish Government plans for public engagement.Cluster 7Supporting Scotland's Climate Change Adaptation Plan (3), including through stakeholder mapping, workshop design and facilitation, briefing papers, evidence and case studies.Number of interviews, case study recommendations and contacts for statutory Adaptation Programme annual progress reports.

  • Documented criteria for those most vulnerable to climate change.
  • Use of appropriate engagement methods engaging those most vulnerable to climate change.
  • Representation of climate vulnerable groups in AS activities.
  • Policymaker feedback on usefulness of these insights and examples where policy language or actions changed.

P

P

P

P

Contact

Email: Rachael.Punton@gov.scot

Back to top