No One Left Behind: evaluability assessment (national impact and economic evaluation)

9. Annex 3: Interview guide

9.1 Central policy and LEP representative discussion guide

9.1.1 Warm-up

Could you briefly describe your current involvement in No One Left Behind?

Follow-up: Which aspects of the approach are you most directly involved with?

Follow-up: Has your involvement changed over time? If so, how?

Can you describe any challenges or successes you’ve observed in your area of involvement?

9.1.2 Impact evaluation priorities

Are there any priority questions you would like a Scotland-level impact evaluation to answer?

Follow-up: What makes these important?

Which factors do you consider most critical for determining the success of No One Left Behind at the Scotland level?

Follow-up: How do you interpret each of these factors in practice for No One Left Behind’s success at a Scotland-level, as opposed to at Local Authority level? Specifically, what should we be looking for in each: for example, what does ‘improving partnerships,’ ‘better service reach,’ or ‘reducing barriers’ look like in practice?

Focusing on service user outcomes, apart from employment outcomes, RAG members also highlighted additional impacts related to health and wellbeing. To what extent do you feel a Scotland-level impact evaluation should consider this?

Follow-up: Beyond employment, health, and wellbeing, are there any other outcomes that should be considered in a Scotland-level impact evaluation?

As part of our assessment, we’re exploring what approaches could be adopted to help understand No One Left Behind’s Scotland-level impact on service users, over and above what would have happened without it - we refer to this as the ‘counterfactual’ scenario. We’re interested in whether you have any views on which scenario is most relevant for a Scotland-level evaluation. Which of these scenarios do you think is more relevant to consider? Could you describe how this would work in practice? For example, how would your organisation operate?

Scenario A: What would have happened under the previous national system?

Scenario B: What would have happened if the Scottish Government had not provided any funding for employment support at all?

What differences in service user outcomes would you expect to see compared to the scenario you selected?

9.1.3 Economic evaluation priorities

From your perspective, what outcomes are most important to consider in assessing VfM at a Scotland-level?

Follow-up: Note to interviewer: Outcomes can include employment progression, earnings, skills development, or wider social and economic impacts.

Follow-up: Are there outcomes that are harder to measure but still important?

Over what period of time should the VfM for No One Left Behind be assessed? Is the 12-month period suggested by the RAG an appropriate point to measure this?

Follow-up: Are there key milestones or policy timelines that should guide this (e.g. funding cycles, reporting deadlines, or programme phases)?

[Asked only to central policy representatives] Does the Scottish Government hold data on planned expenditure, realised expenditure, or both?

[Asked only to LEP representatives]: What cost data does your organisation hold on SG-funded employability activities?

Follow-up: What additional cost data is available for non-SG-funded No One Left Behind activities?

Follow-up: What level of disaggregation is available (e.g. is it available by funding strand (No One Left Behind)? Is it available by priority group?)?

Follow-up: How often is this data collected?

Follow-up: Are these cost data recorded separately, or do they overlap with those for other non–SG-funded activities?

Over what period of time should the VfM for No One Left Behind be assessed? Is the 12-month period suggested by the RAG an appropriate point to measure this?

Follow-up: Are there key milestones or policy timelines that should guide this (e.g. funding cycles, reporting deadlines, or programme phases)?

9.1.4 Additional data points

Are you aware of any datasets (national, local, administrative) that would be relevant to these evaluations and worth considering?

If yes, are there any publicly available sources versus restricted-access sources to note? Can you provide access to restricted-access sources?

Apart from the data collected via the data template, are there any additional data collected and held about participants at the Local Authority/LEP level?

[Asked only to central policy representatives]: Are there any data sources currently being used by the LEPs or LAs that may not be widely known but could support the assessment of the No One Left Behind approach?

Regarding the LEPs/Scottish Local Authorities Economic Development Group’s data toolkit for identifying “who needs support” (SMF Reach 1.1), what is its development status, coverage across LEPs, and likely impact on standardising community needs assessments?

[Asked only to LEP representatives]: Are there any data items held locally that could help illustrate participant activity which are not shared with the Scottish Government?

Follow-up: Are pauses or breaks in participants’ engagement with No One Left Behind-funded work captured in the data?

[Asked only to LEP representatives]: As part of our work, we’re keen to understand the evidence base that practitioners might have access to, including evaluations conducted locally. What is your experience of this, and are you aware of whether there is an evidence-based evaluation used by local partners that might be relevant to a national evaluation?

9.1.5 Understanding the operational reality of No One Left Behind

[Asked only to central policy representatives]: Did the rollout of the No One Left Behind approach take place in all areas at the same time?

[Asked only to central policy representatives]: Is it possible for No One Left Behind referrals to get rejected while satisfying the eligibility criteria?

Follow-up: If yes, under which conditions could this happen?

[Asked only to central policy representatives]: To what extent can information on LEP commissioning approaches (annual investment plans) be collated and compared across LEPs to support a clearer national picture of how No One Left Behind is delivered?

Follow-up: Are there plans to consolidate this information to identify common approaches, gaps, and key variations?

[Asked only to central policy representatives]: Can you identify any patterns in how LEPs deliver interventions?

Follow-up: Are there any practices across LEPs that appear particularly effective?

Follow-up: Are there any identifiable regional variations that stand out?

[Asked only to central policy representatives]: Are there any data item definitions within the data reporting template that you know are interpreted differently in practice? Which ones, and why do you think that happens?

[Asked only to central policy representatives]: For the SMF experience theme, which is not captured in the data reporting template. Are there any known mechanisms in place to report info on that? For example, feedback surveys, service user panels, or any other primary research conducted?

Follow-up: If yes, are there any discussions around standardising these mechanisms? For instance, could all LEPs adopt similar tools?

[Asked only to LEP representatives]: How does the referral process for No One Left Behind applicants work?

Follow-up: Is it possible for No One Left Behind referrals to get rejected while satisfying the eligibility criteria?

Follow-up: If yes, under which conditions could this happen?

[Asked only to LEP representatives]: It has been reported that, at times, participants may pause their involvement in No One Left Behind activities and later resume, sometimes on multiple occasions. Based on your experience, how frequently does this happen?

Follow-up: From the provided data template, we can only see participants’ start and end dates, not what happens in between. Is there a way to capture this, perhaps through an additional data point that could be merged?

Follow-up: Do you record this information locally?

Follow-up: Do you have any suggestions on how this data could be captured to support Scotland-level evaluation activity?

[Asked only to LEP representatives]: Do you have any mechanisms in place to capture participant experience from No One Left Behind-funded activities? For example, feedback surveys, service user panels, or any other primary research conducted?

9.1.6 Debrief

Is there anything else you would like to add that we have not covered in our discussion?

9.2 Data experts

9.2.1 Warm-up

Could you briefly describe your current involvement in No One Left Behind?

Follow-up: Which aspects of the approach are you most directly involved with?

Follow-up: Has your involvement changed over time? If so, how?

Can you describe any challenges or successes you’ve observed in your area of involvement?

9.2.2 Data-related questions:

Beyond the data reported by LEPs through the data reporting template, what other datasets (national, local, or administrative) would be relevant for these evaluations and worth considering?

For each dataset, explore:

How frequently is this information reported, and at what level of detail?

Do you see opportunities to link these datasets with other sources to provide a richer picture of participant outcomes or approach effectiveness? How could this be achieved (e.g. using NI number, name, address)

What would the data access requirements be to allow this material to be used to inform Scotland-level evaluations?

[Asked only to central policy representatives]: What information is currently provided to the Scottish Government through statistical and financial reporting?

Follow-up: Is this disaggregated by funding strand?

Follow-up: Are there any data gaps or robustness concerns we should be aware of?

[Asked only to central policy representatives]: Are you aware of any data items that could help illustrate No One Left Behind activities that are not currently shared as a No One Left Behind reporting requirement? We are interested in data that captures pauses in individuals’ participation.

[Asked only to LEP representatives]: Are there any data items held locally (that could help illustrate participant activity) which are not shared with the Scottish Government?

Follow-up: Are pauses or breaks in participants’ engagement with No One Left Behind-funded work captured in the data? We are interested in understanding who drops out, how often, why, and when.

Follow-up: What would be required to allow these additional data to be used in a Scotland-level evaluation?

[Asked only to LEP representatives]: Regarding the LEPs/Scottish Local Authorities Economic Development Group’s data toolkit for identifying “who needs support” (SMF Reach 1.1), what is its development status, coverage across LEPs, and likely impact on standardising community needs assessments?

[Asked only to LEP representatives]: Are there any additional data that could help us capture the experiences of participants?

Follow-up: Are participants’ experiences captured at the local level? If so, how and at which point(s) in the approach?

[Asked only to LEP representatives]: In addition to the information required in the data reporting template, do you collect any other information on the health or wellbeing of participants? Can we see changes in health at the individual level throughout the course of the approach (for shorter-term conditions)?

[Asked only Improvement Service representatives]: In the LEP Maturity Survey, data sharing and monitoring were highlighted as key areas needing improvement for LEPs. What do you see as the main causes of these challenges, and how could consistency and evidence-based use of data be improved?

9.2.3 Economic evaluation priorities

From your perspective, what outcomes are most important to consider in assessing VfM at a Scotland-level?

Follow-up: Do you expect these outcomes to vary across LEPs depending on the funded activities?

Follow-up: Are there outcomes that are harder to measure but still important?

Over what period of time should the VfM for No One Left Behind be assessed? Is the 12-month period suggested by the RAG an appropriate point to measure this?

[Asked only to central policy representatives]: Does the Scottish Government hold data on planned expenditure, realised expenditure, or both?

Follow-up: Are there any limitations or gaps we should be aware of?

[Asked only to LEP representatives]: What cost data does your organisation hold on SG-funded employability activities?

Follow-up: What level of disaggregation is available (e.g. is it available by funding strand? Is it available by priority group?)?

Follow-up: How often is this data collected?

Follow-up: Are these cost data recorded separately, or do they overlap with those for other non–SG-funded activities?

Do you hold any data on benefit claims of No One Left Behind participants?

9.2.4 Impact evaluation priorities

Are there any priority questions you would like a Scotland-level impact evaluation to answer?

Follow-up: What makes these important?

What factors are most critical for determining No One Left Behind’s success at the Scotland level?

Follow-up: What data is available to measure these in practice?

9.2.5 Understanding the operational reality of No One Left Behind

Did the rollout of the No One Left Behind approach take place in all areas at the same time?

Follow-up: If yes, are there data showing when No One Left Behind was rolled out in each area?

Is it possible for No One Left Behind referrals to get rejected while satisfying the eligibility criteria?

Follow-up: If yes, under which conditions could this happen?

Follow-up: Do you collect any data on rejected referrals? Is it possible to identify these people in your system?

Follow-up: Do you collect any data on individuals who are referred but do not go on to take up support? If so, can these individuals be identified in your system?

Are there any data item definitions within the data reporting template that you know are interpreted differently in practice? Which ones, and why do you think that happens?

[Asked only to central policy representatives]: For the SMF Experience-related questions, are there any plans to standardise the mechanisms used to capture this information required? For example, feedback surveys or service user panels, so that all LEPs could adopt similar approaches?

9.2.6 Debrief

Is there anything else you would like to add that we have not covered in our discussion?