Evaluation of the Adaptation Scotland Programme

D.6: Text from programme management discussion guide

Aims of this interview:

• For staff involved in the management and delivery of the Adaptation Scotland Programme: The primary aim of this interview is to gather information on how the Programme has been delivered in practice, to gather insights for the process element of the evaluation. We also want to understand how well the Adaptation Scotland Programme has been delivered / contributed to its intended outcomes, so the interview gathers some brief insights for the impact element of the evaluation.
• For Scottish Government policy staff: The primary aim is to gather information on whether/how progress towards the intended outcomes of the Programme in Contribution Claims 4 and 5 (see below).

Timings for sections are indicative and will be adjusted by the researcher depending on the individual’s role/responsibilities.

Contribution claims

Contribution claim 4 (CC4): The Adaptation Scotland Programme has contributed to an increase in knowledge, engagement and capacity in shaping and delivering adaptation policy and action in Scotland.

Adaptation Scotland regularly shares the work and findings of the Programme, and insights from wider relevant adaptation work, to its stakeholders. More stakeholders are engaging with the Programme to shape and deliver adaptation policy and action in Scotland. Stakeholders have a greater capacity and feel more empowered to shape and deliver policy and action in Scotland.

Contribution claim 5 (CC5): The Adaptation Scotland Programme has contributed to bettering the value of and acting upon the views of the most vulnerable to climate change at both national and local level.

The Programme has worked with community and third sector organisations to engage groups most vulnerable to climate change and connect them to the networks. Groups most vulnerable to climate change are empowered to share their views and input to effective adaptation resources. Their views are valued and acted upon in adaptation actions at the local and national level.

Introduction and Consent

• Thank participant for taking part (and for their involvement in the evaluation so far if applicable).
• Check that they received and if they have read the information sheet (if they haven’t, explain that you will go through the key points now and can share another copy with them if needed).
• As you may know, Ipsos has been commissioned by the Scottish Government to conduct an evaluation of the Adaptation Scotland Programme. The aim is to assess the impact of the programme, identify what is working well and where improvements could be made in programme delivery.
• Remind them of the purpose of this interview: The purpose of this discussion today is to understand your views on how the Programme has been working, covering both management/delivery processes and the impacts being achieved.
• Explain anonymity and confidentiality: As a reminder - your name will not be disclosed in the reports. However, quotes and elements of reporting may be linked to individual areas. If there is anything that you do not want us to quote you on as part of our reporting, do let us know. I will check in on this at the end of the discussion today.
• The only time that we would disclose your identity is if you told us something that indicated that you or someone else was at risk of harm, in which case we would pass your contact details over to our safeguarding lead.
• Remind participant that there are no right or wrong answers. The purpose of the evaluation is learning, to inform ongoing design and delivery and to understand impacts of the Adaptation Scotland Programme as a whole, and not to evaluate the success of particular organisations or projects.
• Emphasise that taking part is completely voluntary – they don’t have to answer any questions they don’t want to answer and can decide to stop the interview at any point.
• Remind that they are free to change their mind and decide not to take part at any time before or during the interview, or after the interview until the findings have been written up.
• The interview will last approximately 60-90 minutes.
• Check if participant has any questions (offer to go through information sheet if necessary).
• Request permission to record interview. Explain that this is for transcription and analysis purposes and that recordings will not be shared outside the research team at Ipsos.
• If participant does not consent, please take detailed notes.
• If participant provides consent, please start recording.

Their Role

To begin, I’d like to ask about the organisation you work for and your role in the Adaptation Scotland Programme.

Could you start by briefly describing your role at [Scottish Government / Verture]?

• Job title and main responsibilities
• Which team/organisation you work in
• How long you've been in this role

And what is your involvement with the Adaptation Scotland Programme specifically?

• Which cluster(s) do you primarily work on?
• What key activities are you responsible for?
• How do you interact with other team members/partners across the Programme?
• How does your team interact with the [Scottish Government/Verture] teams?

Design of the programme

I’d now like to ask about the design of the Adaptation Scotland Programme and what (if anything) your role is in that. For the purpose of this evaluation, we’re particularly interested in the current contract period (from April 2023-March 2027). (Process)

To your knowledge, who has been involved in designing the Programme and the key activities?

• [if not clear] Have you had any involvement in the design of the Programme, or elements of it?
• [if involved in design] Are there any organisations or individuals whose input would have been beneficial?

[If no design involvement – ask next question then skip to implementation & delivery section]

What is your understanding of the purpose of the Adaptation Scotland Programme? What are the overall aims? (Process)

Can you describe the process for designing Programme activities? (Process)

• How partners contribute to Programme design
• Frequency and format of design discussions
• How decisions are made about what support to offer
• Examples of co-design processes

How satisfied are you with the level of collaboration between those involved in the design of the Programme? (Process)

• What's working well
• Areas where design collaboration could be improved
• Any feedback received from partners about the design process

In your experience, what, if anything, has enabled effective Programme design? (Process)

• Probe if needed: partner capacity, communication, clarity over expectations
• And what barriers, if any, have been faced in the design process?
• Probe: capacity constraints, data-sharing issues, communication, clarity over expectations
• How, if at all, have these impacted the design of the Programme?
• Have these barriers been addressed?

Programme Implementation and Delivery

Moving on now to how the Programme has been delivered across the different clusters. We’ll cover a range of things, including the activities involved, who delivers them, who the target audience is, and progress to date. If I ask about something that doesn’t feel applicable to your area/role, please just say and I can move on.

Activities overview

Researcher check recruitment spreadsheet for cluster relevant to the individual.

Can you briefly walk me through the main activities being delivered under the cluster(s) you are responsible for/involved in? For each relevant cluster, gather high level overview of what is involved (Process):

• Cluster 1 (Public Sector): Engagement events, senior decision-maker targeting, resource development, PSCAN network
• Cluster 2 (Community/Third Sector): Training events, resource development, Climate Action Hubs support
• Cluster 3 (Private Sector): Network scoping, enterprise engagement, tool development
• Cluster 4 (Knowledge/Innovation): Research collaborations, evidence calls, knowledge exchange
• Cluster 5 (Communication): Website optimization, newsletters, case studies
• Cluster 6 (Place-based): Regional partnership support, routemap progression
• Researcher note cluster 7 and cluster 8 covered separately later in interview.

Partnerships

Who is involved in [cluster] delivery? (Process)

• Do you work with any other partners?
• Are there any additional partnerships you’d like to develop? Why?
• [If no] Are there plans to develop work with partners? Why? Why not?

[For those working with partners] Can you tell me more about who these partners are? Probe: Number of partners, types of organisations, their roles in delivery. (Process)

• How were these partners identified and brought on board?
• Are these new or existing partnerships?
• How do you work with them? In what ways do they support? Probe: joint trainings/events, co-authored tools/resources, joint bids/projects

What is your experience of working with these partners? (Process)

• What works well?
• What are the challenges?
• Have you received any feedback from partners about what they feel is working well or could be improved?

Engagement/reach (clusters 1-7)

Who are you primarily trying to reach with Programme activities and outputs? (Process)

• Probe: Target audiences for each cluster, priority groups or sectors, geographic coverage

How do you engage different stakeholder groups? (Process)

• Probe: communication channels, tailoring for different audiences, most/least effective methods

To what extent do you think the intended audiences are being reached? (Process)

• Which groups are engaging well?
• Are there any groups that are harder to reach?
• How could the Programme improve reach with these groups?

Resourcing

Ask all: Is the level of resourcing sufficient to deliver the activities under [cluster] currently? (Process)

• What impact does this have on your ability to deliver the agreed activities?
• Programme Manager only probe on: skills coverage, any staffing or training undertaken or needed

Current progress

How well are these activities progressing? (Process)

• Are you meeting planned outputs/milestones?
• Is there anything that is ahead of/behind schedule?

What are the factors that have enabled delivery of activities under [cluster]? (Process)

• Probes: Partner relationships, clear communication and role definition, data-sharing arrangements that work well, partner capacity that enhances delivery

Are there any barriers or challenges that hinder the delivery activities under [cluster]? (Process)

• Probes: Capacity constraints, data-sharing difficulties, lack of clarity over offer or roles, partner relationships/capacity

Based on what we’ve been discussing so far, to what extent have the activities under [Cluster] been implemented as originally intended overall? (Process)

• Have activities or outputs been delivered according to the initial plans (thinking about budget and time)?
• What adjustments have been made and why?
• Which elements have gone particularly well/less well?
• Any further changes needed to improve implementation?

Is there anything missing from the current Programme? (Process)

• Gaps in provision
• Unmet needs you've identified

Programme Governance and Management

Moving on, I’d like to talk about Programme governance and management. Don’t worry if this isn’t something you know much about, we can move onto the next section.

In your own words, could you explain your understanding of how the Programme governance works? (Process)

• Overall governance structure
• Key decision-making processes
• Your role within the governance structure

What governance documents and structures guide the Programme? (Process)

• How are these documents used in practice?

How do Programme Board meetings function? (Process)

• Frequency of Board meetings
• Who attends and participation levels
• Types of decisions made at Board level
• How Board decisions are communicated and implemented
• Your interaction with the Board

How does the governance structure support Programme and cluster delivery? (Process)

• Ways governance enables your work
• How governance helps coordinate across clusters
• Support for problem-solving and decision-making

What, if anything, works well with the way the way the Programme is governed? (Process)

What, if anything, could be improved about the governance arrangements? (Process)

• Current challenges or constraints
• Gaps in governance structures or processes

Effectiveness in policy making

Now I’d like to talk about the effectiveness of the Programme in terms of wider policy making. Again, if this isn’t something you feel well placed to speak about, please just say and we can move on.

In your view, does the Adaptation Scotland Programme engage with national policy making? In what ways? Probe and seek examples of (Process):

• National boards with AS representation
• Working groups and committees
• Submissions to government consultations on policy development
• Responding to calls for evidence
• Contributing to standards and frameworks
• Use of Programme resources in policy documents

How frequently does this happen?

Who is involved?

How is it decided which areas of policy making the Programme will engage with?

Is the Programme’s involvement too little, too much or about right? (Process)

Are there any policy areas the Programme could engage with more?

How visible is adaptation action across different government policy areas now compared to the previous Programme contract (pre-2023)? (Process)

• Which policy areas show increased adaptation focus
• Has the programme contributed to that?

In your view, to what extent has the Programme contributed to increased ambition in adaptation policy? (Impact)

• Examples of policy ambition changes you've witnessed
• Programme's role in raising adaptation priorities
• Specific policy developments the Programme has influenced
• Comparison with adaptation ambition before the Programme

How is the Programme helping to mainstream adaptation across different policy areas? (Impact)

• Strategies used to integrate adaptation into various policies
• Success in embedding adaptation in sectors like health, transport, planning
• Examples of cross-policy adaptation considerations
• Barriers to mainstreaming and how they're being addressed

Impact

For Scottish Government Policy staff: In this last section we’ll discuss the impact of the Programme: Interviewer share screen with Contribution Claims and introduce these. Make sure to leave time to cover CC4 and CC5. Cover other claims if time.

For other staff: In this last section we’ll briefly discuss the impact of the Programme. We are exploring this in more detail with a range of stakeholders, but wanted to gather your views on this too.

Interviewer share screen with Contribution Claims and introduce these.

Based on your involvement in the Adaptation Scotland programme: Which of these, if any, do you think the Programme has made the most contribution to so far? (Impact)

• What examples of this come to mind?
• Does this vary (e.g. by cluster, stakeholder group, region)?
• Are there any other factors beyond the Programme that could explain this?
• Probes: Legislative and policy drivers (e.g., Climate Change Act, public bodies duties), direct experience of climate change impacts, work of other organisations/networks, participatory processes, knowledge sharing or other engagement campaigns

And which has the Programme made the least contribution to so far? (Impact)

• Why is that?
• Does this vary (e.g. by cluster, stakeholder group, region)?
• Is this happening because of other initiatives?
• See probes above
• What would need to change for the Programme to make more of a difference here?

Summary thoughts

Before we finish, I’d like to gather some summary thoughts on how the Programme has been delivered so far.

What aspects of the Programme are working particularly well?

• Most successful approaches
• Key enablers of success
• Evidence of early or intermediate outcomes you're seeing
• Does this vary depending on: the type of stakeholder? The place?

Are there any aspects could be improved?

• Main challenges or barriers
• Areas needing development

Wrap up and close (2 minutes)

Is there anything else about Programme delivery or impact you'd like to discuss that we haven't covered?

Are there any documents or data you'd recommend we review?

Can I please check if there is anything you would prefer us not to quote or refer to directly in our reports for the Scottish Government?

Thank you very much for your time and insights. Your input will be invaluable for the evaluation.