No One Left Behind and the Young Person's Guarantee: implementation evaluation

Annex A – Discussion guides

Discussion guide for interviews with stakeholders in case study areas

Introduction (3 mins)

Ipsos has been commissioned by the Scottish Government to deliver an implementation evaluation of the development and early delivery of No One Left Behind (NOLB) and the Young Person's Guarantee (YPG). This will assess progress and learning to date, identifying enablers and barriers to implementation. It will consider the extent to which the stated aims and objectives of each are on track to be achieved, and any adjustments that might need to be made in future phases to enable this. The findings will generate valuable insights and learning to inform continuous improvement of employability policy and service design and delivery in Scotland.

As part of this, we are conducting interviews with stakeholders who have been involved in the development and/or implementation of NOLB and YPG. The purpose of these discussions is to deepen our understanding of the context, background and processes involved in the development, early set up and perceptions of progress in implementation.

We will discuss the aspects you feel best placed to comment on, and if you do not have information to answer any questions, please just let me know and we will move on. The interview is expected to last up to 60 minutes.

Findings from these interviews will be used to inform the evaluation of NOLB and YPG which will be published. However, everything you say will be treated in the strictest confidence and findings will be reported in aggregate. No identifying information about individuals will be included in the report, for example, if we would like to quote you, we will do it anonymously. The Scottish Government will not receive notes from individual interviews or attributable comments, but they are aware of who is taking part in these discussions and given the small number of people involved and their roles we cannot guarantee anonymity.

Participation is voluntary and you can change your mind at any time, up until the evaluation report is published.

We would like to record the discussion for analysis purposes. It will not be provided to anyone outside of the evaluation team. The recordings will be securely stored and will be destroyed three months after we have completed the evaluation.

Do I have your permission to record?

Turn on the recorder and record consent to take part and for the discussion to be recorded

Do you have any questions before we begin? Are you happy to proceed?

Background and involvement (5 mins)

Note to interviewers: Once you have established the interviewees' role and involvement in NOLB and / or YPG, please ensure you tailor the subsequent questions accordingly to ensure you are covering both NOLB and YPG equally. This will include exploring for differences between employability services / provision delivered to 16-24 year olds through YPG relative to other age groups under the NOLB umbrella.

To start, can you provide an overview of your role and involvement in NOLB and/or YPG, including how long you have been in this role?

Probe for:

• Strategic responsibilities – involved in discussions / decision-making around purpose, aims and objectives and key design elements
• Set up, development and / or implementation of NOLB / YPG
• Delivery of NOLB / YPG
• Performance monitoring / reporting
• Stakeholder engagement / communications
• Whether role / responsibilities have changed over time

Context & rationale (8 mins)

For the next few questions, I'd like to ask a bit about your understanding of the context to NOLB and YPG and their aims at a national level (we will move onto what it means for your area afterwards).

• What is your understanding of the context and rationale for the introduction of NOLB and YPG at a national level?
• What problem(s) were they seeking to address?
• How would you summarise the key aims of NOLB and YPG?
• What would success for NOLB and YPG look like? What would be the way to define and measure it?
• As far as you know, what would other interested stakeholders consider a success for NOLB and YPG?

What is your understanding / interpretation of the relationship between NOLB and YPG? Probe for whether they view YPG as sitting under the NOLB umbrella or separate.

Now, thinking about your local authority area specifically:

• What are the key employability issues in your area?
• Any key challenges/barriers for specific groups such as those with protected characteristics (including disabled people), those with caring responsibilities and care experienced people?
• Are these different for those aged 16-24 relative to other age groups?
• Any differences compared to Scotland as a whole?

Local approach to NOLB/YPG (8 mins)

Could you please give me a brief overview of how NOLB and YPG work in your local authority to address the key issues that you mentioned?

Change the following section depending on interviewee:

• 1. LA: The Employability Lead, frontline staff involved in day-to-day delivery of employability services or staff involved in planning or provision of employability training
• 2. Third sector
• 3. Employer

LA

• What do you commission and what do you deliver yourselves?
• Do you use well-established partnerships or did you form new ones?
• How many providers are you using?
• What is the profile of those providers? Probe on factors such as: size, geographical coverage, type of organisation (e.g. third sector / private sector), types of activities / support delivered, target beneficiaries.
• Approximately how many employers are engaged in the delivery of employability opportunities/apprenticeships in your area?
• Has it been straightforward to engage employers? What, if any, barriers/issues have come up? How have these been resolved/overcome?
• How have you used Employer Recruitment Incentives (ERI) to create and boost opportunities for young people in your area? How has this worked?
• How has the YPG funding been used in your area? What have been the main categories of spend and approximate percentage split of available funding across each? Has there been any underspend? If so, what are the reasons for this?

Third sector

• How are you involved in the delivery of employability services?
• What services do you offer?
• What are some of the challenges or barriers you face when delivering employability services and how have these been overcome?
• How is the partnership and communication with the LA working?
• Do you work across multiple local authority areas? If so, how does the LA partnership and communication compare across these areas?

Employer:

• How are you involved in delivering services? Apprenticeships?
• Have you used the employer recruitment incentives programme?
• How is the partnership and communication with LA working?

Changes due to NOLB and YPG (8 mins)

Thinking about your approach before NOLB and YPG was introduced:

• To what extent has NOLB been a new approach or opportunity to build on existing services?
• How has the YPG funding impacted on your approach?
• What have been the enablers to change?
• What have been the barriers?
• What are the key lessons learned over the past two years through implementation of the NOLB strategy and YPG?
• Overall, do you feel you are currently delivering against the aims of the NOLB strategy and YPG?
• What do you think of the use of YPG to ensure youth employment during a pandemic / economic crisis? Would you endorse this approach in a similar future crisis?
• How well would you say that the YPG integrated into the wider employability landscape?
• What impact, if any, did the timings of the roll out of the YPG have on implementation and delivery?
• What are the next steps are you focusing on in terms of implementing NOLB/YPG?

Progress in implementation (8 mins)

For the next part of the conversation, I would like to ask about your perceptions of progress in implementation of NOLB/YPG.

Early set up

• How did you experience the early set up, design and implementation of NOLB / YPG?
• How did you engage and communicate with stakeholders/partners/LA?
• Did the LA issue any guidance or have information events? If so, was this helpful and were there any changes you would suggest for future iterations?
• Who was involved at each stage and what was their role?
• What are your perceptions of what worked well / less well in this stage? What are the lessons?

Monitoring and reporting

• What processes are involved in monitoring and reporting on the implementation of NOLB / YPG?
• What performance and monitoring data is collected and by whom?
• How frequently is this data collected?
• Have you encountered any challenges collecting monitoring data? What would help in overcoming these?
• What are your perceptions of what is worked well / less well in relation to monitoring and reporting? What are the lessons?
• If not mentioned: Are you aware of the Shared Measurement Framework for NOLB? If yes, what are your thoughts / feedback on this? How useful or otherwise is it in informing your approach?

Progress on outcomes

• I understand it is early to ask this question, but to what extent do you think there has been progress towards achieving the intended outcomes for NOLB in your area? Probe on: 1) outcomes for service users in terms of progression towards education, employment and training and 2) outcomes for the 'system' of employability support in Scotland relating to better partnership working, more integrated service provision, greater focus on need, etc.

Reaching clients (8 mins)

Promotion

• How is NOLB and YPG promoted to clients in your area?
• What have your experiences been of branding and online information provision?
• What went well / not so well?
• Do they think this has impacted on uptake?

Accessibility and target groups

• What are the key target groups for NOLB and YPG support in your area?
• How easy or difficult has it been for you to engage target groups with NOLB and YPG?
• Are some groups more/less difficult to reach/engage?
• Is demand what you expected it to be?
• Does this differ across different groups? (e.g. young people)
• Are you aware of any specific ways in which services are made accessible to those facing additional barriers? For example:
• Disabled people / people with long term conditions
• Care experienced people
• Minority ethnic groups
• People with convictions
• Young parents
• Families experiencing child poverty
• What has worked well/less well to support these groups to engage with NOLB and YPG?
• Do you think there is anything more that could be done to ensure NOLB / YPG funding is reaching these groups?
• Are there any specific ways in which people with caring responsibilities (including parents) are supported to engage with NOLB and YPG?
• What specific barriers do these service users face?
• What are the challenges in supporting this group?
• What has worked well/less well?
• What more could be done?

In summary,

• Is there anything that could be done to improve the reach of NOLB / YPG in general?

Summary (5 mins)

Thank you, we're coming to the end of the interview now, so I'd like to ask a couple of summary questions.

• Could you please give a score 1-5 (1= very poor, 3=neutral and 5= excellent) of how well you think NOLB has been implemented in your local authority area so far?
• Could you please give a score 1-5 (1= very poor, 3=neutral and 5= excellent) of how well you think YPG has been implemented in your local authority area so so far?
• What would you say is working particularly well in implementing NOLB and YPG in your area?
• And overall, what are the key things that would you say could be working better or any recommendations for the future?

Wrap-up (2-3 mins)

Is there anything we haven't discussed already which you think may be helpful for us to be aware of in delivering the evaluation?

Do you have any questions for me?

Thank you very much for speaking with me today.

Discussion guide for groups with employability staff

Introduction (5 mins)

Introduce Self / Ipsos (independent research company) / Tech Support / Anybody Else.

I'd like to thank you all very much for coming today and offering to contribute your views as part of this research. I'll start by going over some details about the research, as well as what we'll cover in this session and how the discussion will work.

Firstly, a reminder about what the research is about. As you will know, Ipsos has been commissioned by the Scottish Government to deliver an implementation evaluation of the development and early delivery of No One Left Behind (NOLB) and the Young Person's Guarantee (Guarantee). The findings will generate valuable insights and learning to inform continuous improvement of employability policy, service design and delivery in Scotland.

As part of this, we are running group discussions with staff members who have been involved in the implementation/early delivery of No One Left Behind and the Young Person's Guarantee. We will be asking about your views on employability issues in your area and changes as a result of No One Left Behind and the Young Person's Guarantee, including what is working well and what could be improved.

Findings from these discussions will be used to inform the evaluation and a report will be published. However, everything you say will be treated in the strictest confidence. No identifying information about individuals will be included in the report. We may quote something you say, but this would be done anonymously. For this reason, we ask that you do not discuss the views raised today outside of this session.

Run through practicalities / ground rules:

• The discussion will last up to 90 minutes.
• Keep microphones muted if there is background noise.
• Participation is voluntary - can change mind at any time, up until the report is published. This applies to individual questions, completely fine to skip.
• Explain your role – might need to interrupt/move people on to make sure everyone has the chance to speak / so we can cover everything, and finish on time.
• Acknowledge limitations on anonymity of professional stakeholders (e.g. few people in a relevant post) – let us know if anything off the record / not quoted.
• Reminder that we are not evaluating individual services / local authorities, but trying to get an idea of how well implementation is going as a whole, across Scotland. Case study areas will not be named in the report.
• With your permission, we would like to record the discussion today to help with our notes and to ensure we don't miss anything that you share. This would only be an audio recording – not video. The recording would not be shared with anyone outside of the evaluation team except those transcribing the recordings to help with our notes and it would be securely stored and then securely destroyed three months after we have completed the evaluation.

Do I have your permission to record?

Before I start recording, are there any questions about anything I've just said or about the research in general?

Turn on the recorder and record consent to take part and for the discussion to be recorded.

Introductions (10 mins)

To start off with it would be great if everyone could briefly introduce themselves.

Can you please tell us about the organisation you work for, the types of services you deliver and how long you've been working in employability services.

Awareness and understanding of NOLB / YPG (10 mins)

I'd now like to ask some questions about your understanding / interpretation of the key aims and objectives of NOLB and the Guarantee.

How would you summarise the aims and objectives of the NOLB strategy?

• Why was NOLB introduced? What 'problem' was it seeking to address?
• What would success for NOLB look like? How could this be defined/measured?

How would you summarise the aims and objectives of the Guarantee?

• Why was the Guarantee introduced? What 'problem' was it seeking to address?
• What would success for the Guarantee look like? How would this be defined/measure?

To what extent do you feel the aims and objectives of NOLB and the Guarantee been clearly communicated to you?

• How were they communicated?
• What has worked well / less well in terms of communications relating to NOLB and the Guarantee? Do you have any suggestions for improvement?

Are you familiar with the NOLB Customer Charter and Service Standards?

• Do you have any feedback on these?
• How and in what ways (if at all) are they being used to inform and support your work?
• How do you know if you are meeting the aims and principles of the Customer Charter and Service Standards? What (if any) monitoring or data / evidence gathering is being done in relation to these?
• Are you familiar with the Shared Measurement Framework?

Local approach to employability services (10 mins)

What are the key employability issues / challenges in your local authority area?

• Any challenges/barriers faced by specific groups, such as those with protected characteristics, caring responsibilities or who are care experienced?
• Any differences in the employability issues faced by those aged 16-24 compared to other age groups? What do you think are the reasons for those differences?
• Any issues specific to your local area (i.e. different to Scotland as a whole)?
• Have these issues changed over the time and since NOLB and the Guarantee were introduced? What has been driving these changes?

Implemen4tation of NOLB (15 mins)

The next questions are about what, if anything, has changed about how employability services are designed and delivered in your area since NOLB was introduced in 2019.

What (if anything) is new or different about employability services are delivered under NOLB relative to previously?

• What has changed and why?
• How far do these changes represent an improvement to delivery of employability services in your area? Why?
• Any specific examples of changes introduced and the difference this has made?

What have been the barriers / challenges faced in the early operationalisation and implementation of NOLB / the new Scottish approach to employability in your area?

• How can these barriers / challenges be overcome?
• What is the role of local, regional and national stakeholders in addressing these?

To what extent do you think employability services in your area are being delivered in line with the principles / values set out in the NOLB Customer Charter and Service Standards? Why do you think that?

• Note to facilitators: have the Customer Charter / Service Standards to hand for reference or to quote for those who are not familiar with these.
• What monitoring or evidence gathering do you do in relation to the CC and SS?

What are the lessons learned from early implementation of NOLB that could be used to inform future phases?

Implementation of the Young Person's Guarantee (15 mins)

What has been the impact of the Guarantee on employability services in your area?

• How has the funding been used? (What services / activities have been delivered through the Guarantee and by whom?)
• What is new / different about the Guarantee compared to previous employability support available to young people (under the age of 25) in your area?

How were these decisions made?

Have you used Employer Recruitment Incentives to create opportunities for young people in your area?

• Why / why not?
• If yes: what approach has been taken to this? How well or otherwise has this worked?
• How successful have ERIs been in helping young people to progress into employment? How (if at all) has this been monitored?

What monitoring and/or evaluation has been taking place relating to the YPG generally?

• How well or otherwise is this working?
• What is going well / less well and why?

What have been the enablers to implementation of the Guarantee?

• What has worked well and why?
• What have been the success factors?
• What are the lessons?

What have been the barriers / challenges to implementation of the Guarantee?

• How can these barriers / challenges be overcome?
• What is the role of local, regional and national stakeholders in addressing these?

How did the pace of roll out of the Guarantee impact on implementation and delivery? What are the lessons from this?

• What worked well / could have been done better?

Do you think the Guarantee was the right response to mitigating the impact of the pandemic on young people's progression in learning and work?

• Why do you think that?
• What else / more do you think could have been done?

What are the lessons learned from implementation of the Guarantee that could be used to inform future phases?

Reaching participants / service users (15 mins)

Promotion

How are participants referred to / made aware of the employment support services available in your area?

• What are the main referral routes / partners?
• How well or otherwise are these working?
• Do you feel the service is reaching those people who need it most? Why do you think that?
• How well do you feel employability and other services in your area work together to support participants?

Accessibility and target groups

Who are the key target groups for employability support in your area?

• How did you identify these groups?
• Have these changed over time / since NOLB and the Guarantee were launched?
• How and in what ways? What has driven this change?

How easy or difficult has it been for you to engage these target groups?

• Are some groups more/less difficult to reach/engage? Why is that? What more could be done to reach those groups?

Is demand what you expected it to be?

• Does this differ across different groups? (e.g. young people)

Are there any specific ways in which services in your area are made accessible to those facing additional barriers?

E.g.

• Disabled people / people with long term conditions
• Care experienced people
• Minority ethnic groups
• People with convictions
• Parents

What has worked well/less well to support these groups to engage with NOLB / YPG?

• Do you think there is anything more that could be done to ensure NOLB / YPG services are reaching these groups?

Are there any specific ways in which people with caring responsibilities (including parents) are supported to engage with NOLB and YPG?

• What specific barriers do these service users face?
• What are the challenges in supporting this group?
• What has worked well/less well?
• What more could be done?

How might services be made more inclusive of underrepresented groups?

Recommendations (5 mins)

What (if anything) do you think could be done to further develop or improve employability services in your local authority area?

Is there anything that you think could be done nationally to further develop or improve employability services in Scotland?

What do you think the future priorities for employability services in Scotland should be?

Wrap up (2 mins)

That brings me to the end of my questions now, but is there anything I may have missed that you think may be helpful for us to be aware of?

Do you have any questions for me?

Thank all for attending the discussion.

If any further questions – do get in touch via email.

End recording.

Discussion guide for interviews with service users

Introduction (3 mins)

Note: researchers should have service user survey responses to hand

Introduce self and Ipsos

Thank you for completing the online survey about your experiences of accessing employment services and for agreeing for us to contact you to ask some follow up questions.

Introduce the research: Ipsos has been appointed by the Scottish Government to evaluate employment support services in Scotland. As part of this research, we are conducting interviews with people who have accessed services.

The interview will last up to 20 minutes. You will get £30 as a thank you, paid as a voucher.

Provide reassurances of anonymity. No identifying information about individuals will be passed to anyone outside the Ipsos research team. If we would like to quote you, we will do so anonymously. It will not be possible for the Scottish Government or anyone else to identify individuals in any of the reports that Ipsos produce.

No one at the Scottish Government, or in the local authority or organisation through which you access employment support services will know who took part in the research so taking part will not impact any current or future services you might use.

Taking part is voluntary, you can change your mind at any time, up until the evaluation report is published. You don't have to answer any questions you don't want to and we can stop the interview at any time.

Request permission to record interview. Explain that this is for transcription and analysis purposes and that recordings will not be shared outside the research team at Ipsos. The recording will be securely stored and deleted three months after the research finishes.

Check if any questions

Turn on the recorder and record consent to take part and for the discussion to be recorded

Background (5 mins)

To start off with could you tell me about your employment journey so far?

If unclear: can you please confirm what you have been doing for the last 12 months? Are you currently in education, training or employment?

Accessing services (5 mins)

You said in the survey it was [easy/difficult/neither nor] for you to find and use employment support services. Can you say why that was the case?

What worked well?

What were the challenges? Barriers? What would have made the experience better for you?

Is your access to employment support services affected by any health needs or caring responsibilities?

Support received (5 mins)

You said in the survey that you were [satisfied/dissatisfied/neither nor] with the services you received. Can you tell me a bit more about why you felt that way? Probe:

• What their expectations were
• What they liked / disliked
• Was it due to the way it was delivered (F2F, group, online)
• How was your relationship with your key worker? What worked well/could have been improved?
• Did you feel treated with dignity and respect? Could you please give an example?

Are you still receiving support? If so, what type of support?

Is there anything that would have made you feel more satisfied with the services or would have helped your journey towards finding work?

Did you also receive any other support, such as childcare or transport, to help you with your search for training or employment?

Outcomes (5 mins)

How, do you feel you the services you accessed benefited you, if at all?

What was it about the support you received that made the most difference?

Do you think the support you received will help you in future? Why do you think that?

What education and employment opportunities do you feel are available to you now?

What education and employment opportunities would you like to have going forward?

Recommendations (3 mins)

Do you have any recommendations for how the employment services available to you could be further improved in future? Probe on:

• Reasons for any recommendations suggested
• What difference these could make to them and others

Wrap up (2 mins)

Is there anything else you would like to say?

Before we finish, we have another paid research opportunity that I think you would be a good fit for. It involves completing some tasks on an app called Ipsos App Life, it's an online diary to share how you're feeling in the moment, for example by sharing photos, short video clips or sound clips, as well as text. There will be 5 tasks over 2 weeks and they will be about how you're feeling about your employability journey. We are giving people £50 as a thank you for taking part in the stage of the research. If you are interested in finding out more, I can share our information sheet with more details. Would you like to receive more information about this? Yes/no?

We have also been asked by The Scottish Government if we could share your contact details with them so they can contact you about the possibility of being involved in a service user panel. Do you give permission for your contact details to be shared with The Scottish Government? Yes/no?

Switch off recorder. Explain next steps, collect details for incentive, thank and close.