No One Left Behind: evaluability assessment (national impact and economic evaluation)

5. Findings

This chapter presents the findings from activities outlined in the previous chapter, drawing on stakeholder interviews, the data mapping and assessment exercise, and the refined ToC. It is structured to reflect each of these components in turn, before being considered together in the following chapter.

5.1 Interview findings

Interview findings are organised thematically under three areas: (i) understanding the operational reality of No One Left Behind; (ii) considerations for a future evaluation; and (iii) data-related issues. The purpose of exploring the first thematic area was to validate and expand our understanding of No One Left Behind operations, rather than assess what works and what does not, which would fall under the scope of a process evaluation. As a result, findings for the first thematic area are presented in Annex 1: Operational reality interview findings, while the remaining findings are included in the sections below.

Although different stakeholder types were engaged, their views often overlapped and addressed similar themes. For consistency, findings are therefore analysed and presented together, with stakeholder type only indicated where views were specific to a particular group. The stakeholder typology used is: (i) central policy representatives^[3]; (ii) LEP representatives; and (iii) data experts.

Findings are presented in the order of the most frequently raised themes within each topic. However, due to the small number of interviews, quantification (e.g. “most”, “many”, “few”) is not appropriate. Where views diverged, both perspectives are reported.

5.1.1 Evaluation scope

Questions in this thematic area explored how a Scotland-level impact and economic evaluation of No One Left Behind should be designed and structured. This included service user outcomes, potential areas for monetisation in a VfM assessment, and perspectives on the appropriate evaluation timeframe and the most suitable counterfactual.

Suggestions for a national impact and economic evaluation of No One Left Behind

Stakeholders noted several important considerations regarding the feasibility and scope of a national impact evaluation of No One Left Behind. A recurring concern was that the significant variation in delivery models across LEPs limits the extent to which outcomes can be meaningfully compared at the Scotland level. As a result, several interviewees questioned whether a single national-level evaluation would risk oversimplifying context-specific outcomes.

There was also strong advocacy for a complementary process evaluation, with stakeholders emphasising that understanding how and why implementation differs across areas is essential for interpreting the observed impacts. In this context, stakeholders believed that these outcomes should be considered within the broader ambition of No One Left Behind as a system-change intervention rather than as standalone measures of success. In particular, they highlighted the importance of assessing whether the approach has strengthened integration across employability services, reduced duplication, and enabled more coherent, person-centred support compared to the previous model.

However, the above considerations fall outside the scope of the present study, which focuses solely on the feasibility of conducting impact and economic evaluations. Among the stakeholders who suggested impact evaluation priorities, the research questions focused on service user outcomes, examining the extent to which the service improves users’ experiences, and on longer-term outcomes. The following questions were highlighted:

• How effective is No One Left Behind at reaching those further away from the labour market?
• To what extent do service users move into employment, gain accredited qualifications, or progress into further education or training?
• To what extent are Local Employability Partnerships producing outcomes for SG priority groups?
• Do No One Left Behind services improve efficiency and user satisfaction compared to the previous national model?
• How does variability in LEP implementation affect outcomes, and which factors drive success?
• To what extent has No One Left Behind driven systemic change in employability services and users’ outcomes compared to the previous model?

5.1.2 Service user outcomes – deep dive

Stakeholders identified a wide range of outcomes for service users, recognising No One Left Behind’s wide-ranging impact. Most outcomes mentioned centred around employability, while others included reach, progression through employability pathways, wider personal circumstances, and longer-term social and economic impacts. Below, we describe the most commonly mentioned outcomes in more detail.

Employability outcomes remain the central measure of success, with both central policy and LEP representatives consistently emphasising their importance. Alongside employment, outcomes such as gaining qualifications, progressing into further or higher education, and engaging in volunteering were seen as important indicators of progress in the employability journey. However, stakeholders also noted that these measures do not fully capture the contribution of No One Left Behind, particularly around softer employability outcomes such as improved readiness for work, increased confidence, and regular engagement with services or activities

Service reach refers to the extent to which No One Left Behind engages priority and underserved groups. While LEP representatives reported monitoring engagement with these groups locally, this information is not intended for national reporting purposes. Central policy representatives noted that reach is currently best understood at the local level through LEP AIPs, which set out intended targeting approaches and resource allocation, rather than actual provision delivered in practice. They also highlighted a broader ongoing challenge in assessing whether all those in need are being identified and supported, particularly where data on unmet need remains limited.

Health and wellbeing outcomes were frequently highlighted as important in their own right. Interviewees noted that many participants experience mental health challenges, anxiety, or other long-standing health conditions, and that access to appropriate support can improve confidence, resilience, and overall wellbeing. However, these outcomes are difficult to measure consistently and are not systematically captured in a way that allows for clear attribution to specific interventions.

Income maximisation was also identified as an important outcome, particularly in relation to supporting individuals to increase household income and reduce material deprivation. This includes assistance with welfare benefit applications and broader financial support to improve living conditions, such as addressing essential household needs or resolving basic security issues.

In addition, interviewees highlighted a range of broader outcomes that extend beyond immediate employment gains. These include avoiding prison sentences, reducing reoffending, contributing to tackling child poverty, and supporting parents into work. Reduced reliance on welfare was also noted as an outcome associated with sustained employment and improved financial independence.

5.1.3 Monetisation of outcomes

Building on the broader range of service user outcomes identified, interviewees were also asked specifically about which of these could realistically be monetised or incorporated into a VfM assessment. In this context, stakeholders identified a range of potential outcomes for VfM analysis, noting that employment-related and intermediate outcomes are generally more straightforward to measure and monetise than broader wellbeing or social impacts.

• Employment outcomes: Job entry, sustained work, progression, apprenticeships, self-employment, or further education/training.
• Income maximisation: Increased earnings and improved household finances via work, benefits, and reduced material deprivation.
• Health and wellbeing: Reduced hospital visits could proxy health outcomes, reduced anxiety and stress.
• Reduced reliance on welfare: Fewer out-of-work benefit claims.
• Business impacts: Employer recruitment support, staff retention, business sustainability, and growth.
• Wider social outcomes: Reduced reoffending, avoided prison costs, tackling child poverty, and gains from supporting parents into work.
• Wider economic benefits: Increased tax revenues and higher consumer spending.
• Intermediate outcomes: participants supported, and softer progress measures where these can be robustly evidenced.

5.1.4 Appropriate timeframe and counterfactual scenario for assessing VfM

On the appropriate timeframe for assessing No One Left Behind’s VfM, stakeholders expressed differing views. One perspective was that a 12-month period provides a pragmatic and workable basis for analysis, as it aligns with existing reporting structures and allows for the assessment of overall achievements within a clearly defined timeframe, regardless of when individuals entered the programme. In contrast, others argued that a longer timeframe, around two to three years, would be more appropriate to fully capture the effects of the intervention. From this perspective, a longer horizon is needed for outcomes to materialise, particularly regarding sustained employment and potential longer-term impacts on health and wellbeing.

With regard to the scenario against which No One Left Behind should be compared (counterfactual), stakeholders generally preferred to use the previous nationally commissioned employability programmes as the counterfactual, rather than assuming a scenario with no Scottish Government support. Earlier programmes such as Fair Start Scotland were seen as a more meaningful comparator, as they provided a consistent national framework and a more standardised delivery model. In principle, this offered a clearer basis for understanding the shift towards No One Left Behind’s more locally driven and integrated funding approach. A “no support” counterfactual was raised only as a theoretical reference point, but was considered difficult to operationalise in practice.

In addition, stakeholders emphasised that even comparisons with the previous system are complex, as earlier Scottish Government employability programmes (Fair Start Scotland, Community Jobs Scotland, and the Employability Fund) often overlapped and were not always designed to operate as a single coherent system. More broadly, variation in how LEPs allocate funding and target different groups further limits the extent to which a clear and consistent national counterfactual can be established.

A more detailed discussion of counterfactual approaches and their robustness is provided in the following chapter under the Counterfactual scenario & control groups section.

5.1.5 Evidence base for an impact and economic evaluation

The last topic discussed with stakeholders was the availability of the evidence base, including quality-of-service user data reported to the Scottish Government via the DRT, as well as considerations regarding the availability and robustness of cost data.

Service user data

The DRT was often cited as the primary source of evidence regarding service user information submitted to the Scottish Government. Stakeholders described the DRT as a comprehensive dataset with robust quality assurance mechanisms. However, over the course of discussions, stakeholders also identified several key challenges associated with relying on DRT data as the sole source of service-user evidence for evaluation. Specifically, they highlighted challenges relating to:

The interpretation of definitions and questions: The DRT can be populated differently across LEPs due to varying interpretations of definitions and the level of detail required, which is believed to lead to inconsistencies in reporting. In this context, central policy stakeholders informed us that the SMF working group is conducting a survey of the DRT to gather feedback and improve it in the near future.

Reliability of information: The DRT relies on self-reporting, which may affect data reliability, particularly for financial information such as rates of pay. Language barriers can also limit individuals’ ability to provide accurate information, access support, or be effectively targeted by LEPs. For example, LEP representatives reported difficulties identifying Ukrainian service users and determining the resources required to support them effectively at the local level.

Capturing repeated engagement with services: Stakeholders highlighted that repeat participation is not recorded as new entries in the DRT. LEP representatives estimated a high reengagement rate (around 30%). This not only leads to an underrepresentation of delivery activity in aggregate statistics but also results in changes in circumstances or newly achieved outcomes that are not consistently captured. For example, life events such as becoming a parent are not reflected in the initial record. In addition, the DRT does not record reasons for drop-out or reengagement. LEP representatives reported that, while such information is held locally within management information systems, it is not reflected in the national dataset.

Collection over time: Stakeholders noted that some data fields of the current DRT were not included in its first iterations. The template was described as an evolving tool that has developed alongside changes in the approach’s priorities and funding structure. For example, disability- and parental-related information was added following the introduction of the respective funding strands.

Data lag: Stakeholders noted that there can be delays of several months before participant data is entered into the system, meaning individuals may receive support for some time before appearing in official records. For example, someone supported in quarter one may not be included until quarter three, so recent activity may not be fully reflected in current reports. Stakeholders also noted that frequent updates to the reporting template, combined with changing funding streams, particularly during COVID and the introduction of the Young Persons Guarantee and parental support funding, have added further complexity to maintaining data consistency.

Limited coverage of service experience and “soft outcomes”: Stakeholders noted that there is insufficient data on outcomes such as health and wellbeing, while service user experience is not captured consistently.

Upcoming developments

Reflecting on the aforementioned challenges, discussions with data experts and central policy stakeholders identified several developments to improve data collection and robustness. In particular, discussions focused on:

• Addressing limitations in DRT-reported information: A significant development relates to potentially linking No One Left Behind participant records with HMRC data. Stakeholders informed us that a legal gateway has now been established. This would provide more robust information on employment outcomes, including job type, sector, earnings, contract status, and longer-term employment trajectories. Furthermore, the HMRC linkage could support tracking beyond the current 12-month follow-up period and reduce reliance on self-reported data in the DRT.
• Understanding unmet need and service reach: Stakeholders also highlighted the potential value of linking DRT to DWP data. This could provide insight into benefit receipts, eligibility for No One Left Behind services, and individuals who may benefit from support but are not currently engaging. In turn, this would strengthen understanding of unmet need and help identify gaps in service reach. Any further linkage work, including with DWP or health datasets, would require additional legal, governance, and data protection processes.
• Reengagement tracking improvements: Stakeholders noted that work is currently underway to improve how repeat participation is recorded through a dedicated “restarts” project, which aims to capture both initial and updated characteristics each time a participant re-engages with the service.
• Capturing service user experience: SG has commissioned a national evaluation of user experience. This includes both surveys and in-depth interviews to understand how people experience the service, why they stay engaged, and, to the extent possible, why some people drop out.

Availability of cost data for an economic evaluation

Stakeholders also discussed the availability and robustness of cost data for No One Left Behind. Financial information for No One Left Behind is reflected primarily in the total grant value allocated to LEPs, which indicates annual spending across areas. The Scottish Government maintains an overall estimate of total investment, noting that it is not completely accurate. LEPs also record financial information through AIPs and quarterly financial returns, covering participant and staff costs as well as wider operational expenditure. A breakdown of expenditure is available for the last three years (i.e. from 2023 onwards). While this provides an overall view of spend, stakeholders highlighted several important considerations when examining financial information. These relate to:

• Disentangling funding between No One Left Behind and other sources: Although LEP reporting distinguishes between No One Left Behind funding and other sources, stakeholders emphasised that in practice, services are delivered through blended funding models. Individual support packages are often co-funded by No One Left Behind funding, Local Authority budgets, and UK Government programmes, making it difficult to attribute expenditure or outcomes to a single funding stream. The proportion of funding from No One Left Behind varies by area, ranging from around 50% to 90%, depending on local budgets and additional funding streams. As a result, the reported split should be interpreted as an indicative estimate rather than a precise allocation, particularly when comparing across funding sources.
• Disentangling spending and impact across No One Left Behind funding strands: AIPs also set out budgets across cohorts and priority groups, but delivery is not strictly segmented in practice. Services frequently support mixed participant groups, making it difficult to directly and consistently attribute spending to specific funding strands. This is further complicated by variation across areas, the relatively recent introduction of the Specialist Employability Support funding strand, and overlaps in group classifications, in which individuals may fall under multiple categories.

5.2 Data mapping and assessment findings

This section presents the datasets included in the final, refined shortlist recommended for the No One Left Behind evaluation. It outlines the key characteristics of each dataset, including their main strengths and limitations, to support a clear understanding of their suitability for the evaluation.

5.2.1 Shortlist of data sources

A. HMRC Real Time Information (RTI)

Description: The HMRC RTI dataset is a comprehensive administrative source that records employee pay as reported by employers during each pay period. It is used for tax administration and labour market analysis. This dataset is a valuable source of information because it provides accurate, high-coverage data on employees within the UK tax system.

Strengths: In the context of No One Left Behind interventions, it enables reliable identification of individuals’ employment outcomes, including job type, sector, earnings, contract status, and longer-term employment trajectories. This includes outcomes for both No One Left Behind participants and the general population for comparison.

Limitations: HMRC RTI includes only individuals paid through the Pay As You Earn (PAYE) system, so self-employed individuals are not captured. Additionally, it does not contain information on broader outcomes such as education, health, or wellbeing. These gaps can be addressed to some extent by linking the dataset to other administrative sources using NI numbers, which are available within it. Finally, at the time of writing, discussions with HMRC cover only No One Left Behind participants, which means that the RTI cannot be used for the identification of a control group.

B. Data provided by the Local Authorities in Scotland via the DRT

Description: Data provided by local authorities in Scotland through the DRT consists of quarterly monitoring information submitted to the Scottish Government. Our own review and discussions with stakeholders suggested that the DRT covers each No One Left Behind service user across a range of data fields, including personal characteristics and outcomes such as qualifications achieved, employment entry and sustainment, and job type. The maximum number of data fields is 105, assuming all follow-up fields for every outcome are completed. In practice, the number of fields completed is lower.

Strengths: The DRT offers rich information on No One Left Behind participants, with data experts noting that there are currently around 99,000 individual participant records spanning the last seven years (from 2019). Updates are submitted quarterly and subject to repeated validation checks, which contribute to the overall robustness of the data. At the same time, the Data and Reporting Group carries out ongoing quality assurance and refinement processes. Most notably, the DRT is the only robust way to identify No One Left Behind participants, who can then be tracked in other datasets and excluded from the control group (see the Counterfactual section for more details).

Limitations: Despite its strengths, there are concerns about the DRT’s data quality and consistency. As noted by stakeholders and detailed in the Interview Findings section above, DRT data is subject to: (i) inconsistent interpretation of definitions and data fields by the staff completing it; (ii) limitations related to self-reporting of data and language barriers; (iii) missing data on reengagement; (iv) varying data fields across years; (v) data lag between a participant receiving support and being reflected in the data; and (vi) limited coverage of user experience and other “soft outcomes”.

C. Registration and Population Interaction database (RAPID)

Description: RAPID is a large administrative dataset for the UK, created by linking benefit records from the DWP, LA housing benefit systems, and earnings from HMRC PAYE. It provides an annual overview of individuals’ employment, benefit receipt, and pension activity, along with 200 additional variables.

Strengths: RAPID’s main use in this evaluation is to provide a comprehensive list of Scotland benefit claimants and the amounts claimed. It uses NI numbers, enabling seamless linkage with the DRT and the HMRCRTI datasets. This is key for identifying potential service users with unmet needs (i.e. benefit claimants who are not No One Left Behind participants) and for enabling the identification of a comparable control group.

Limitations: It does not include certain key employment details, such as the sector of employment or the number of hours worked. Additionally, information on education outcomes is not available unless the dataset is linked with other sources. While RAPID uses NI, linking it to the DRT requires appropriate legal and governance agreements, which may affect the feasibility and timing of the analysis.

D. Labour Force Survey (LFS)

Description: The LFS is a large, quarterly household survey conducted by the ONS. It gathers detailed information on individuals’ employment circumstances, including their work status and occupation across the UK.

Strengths: The LFS datasets include important information on health-related indicators, such as (i) long-term health conditions; (ii) disability status; (iii) type of conditions; (iv) impact on employment; and (v) temporary sickness. This type of health information is not captured in most other datasets.

Limitations: The LFS is a survey, and not an administrative dataset. As a result, it does not cover the entire population. The sample size for Scotland is relatively small, which can reduce the representativeness and robustness of the findings. In addition, the dataset does not include universal identifiers, such as NI numbers. This makes it difficult to link LFS data with administrative datasets or with Local Authority records on No One Left Behind participants, limiting its use for individual-level analysis.

E. Annual Population Survey (APS)

Description: The APS is a key UK dataset that provides detailed information on population characteristics and labour market conditions. It is not a standalone survey; instead, it combines data from multiple waves of the LFS with additional local boost samples, which are designed to improve coverage and reliability at the regional and local levels.

Strengths: The APS is included because it can be used to analyse a comparable group of individuals who have not participated in No One Left Behind services. Its larger sample size, compared to the LFS, has historically made it particularly useful for producing more detailed local and regional labour market statistics.

Limitations: Although the APS has benefited from increased sample sizes due to boost samples, these boosts are being discontinued from 2026 onwards. This change is expected to reduce the overall sample size. Consequently, this will limit the level of detail and reliability of local and regional labour market analysis in the future.

Other specific details regarding how these datasets perform against the data assessment framework are presented in the table below. We rank each dataset on a RAG scale (i.e., red, amber, green) based on its coverage of the SMF. In this context, green colouring indicates satisfactory coverage, amber indicates partial coverage, while red indicates little to no coverage.

Dataset	Geographical coverage	Frequency	Aggregation level	Reach	Progression	Skills Assign-ment	Value of services	Conclusion
HMRCRTI	UK with national breakdowns	Monthly	Individual	Age, sex, NI number	Employment status, job continuity		Gross pay, net pay	Should be used to measure employment-related outcomes from both No One Left Behind participants and the control group
Ranking	Green	Green	Green	Amber	Green	Red	Green	Green
Data reporting template for LA	Scotland	Quarterly	Individual	Sex, DOB, religion, ethnicity, NI number, childcare	Employment and education data tracked over several weeks		Rate of pay	Should be used to identify No One Left Behind participants
Ranking	Green	Green	Green	Green	Green	Red	Amber	Green
RAPID Dataset	UK with national breakdowns	Annually	Individual	Age, sex, NI number, benefit claims, children in the household and health conditions, to the extent that related benefits are claimed	Employment, self-employment status		Annual earnings	Should be used to balance treatment and control groups based on a wider set of variables than available in RTI data.
Ranking	Green	Amber	Green	Amber	Amber	Red	Amber	Green
Annual Population Survey (LFS Wave 1 + 5 boosts)	(i) UK with national breakdowns (ii) Additional coverage compared to LFS	Annually	Individual	Age, sex, religion, ethnicity, dependent children, health, disabilities	Employment, self-employment status, apprenticeships, type of contract, education, employer-offered training/education	Basic skills - literacy, numeracy, core skills, skills for work, certifications	Hourly pay, weekly pay	Should be used to get health and education related information for No One Left Behind participants and the control group (similar to LFS, but with a larger sample size)
Ranking	Green	Amber	Green	Green	Green	Green	Green	Green
Labour Force Survey (LFS)	(i) UK with national breakdowns (ii) Concerns regarding Scotland sample size	Quarterly	Individual	Age, sex, religion, ethnicity, dependent children, health, disabilities	Employment, self-employment status, apprenticeships, type of contract, education, employer-offered training/education	Basic skills - literacy, numeracy, core skills, skills for work, certifications	Hourly pay, weekly pay	Superseded by Annual Population Survey
Ranking	Amber	Green	Green	Green	Green	Green	Green	Amber

5.3 Refined Theory of Change (ToC)

5.3.1 About the Theory of Change (ToC) framework

A ToC framework demonstrates how a policy, intervention, or programme leads to change. It maps how inputs and activities lead to immediate outputs, which in turn contribute to outcomes and long-term impacts aligned with policy objectives. In this analysis, the ToC illustrates how No One Left Behind is intended to achieve its goals and how change is expected to occur through the approach.

As detailed in the Methodology section, we used our desk-based review and stakeholder engagement to revise the existing logic model into a refined ToC that maps how No One Left Behind operates and creates impacts. The revised ToC is centred on service user outcomes and structured around the three No One Left Behind funding strands (All-Age Employability Service, Parental Employability Support, and Specialist Employability Support), providing a framework with a stronger focus on LEP service delivery and the mechanisms through which outcomes are generated.

The ToC, detailed in the next section, is designed to serve as a central reference point for any future evaluation of No One Left Behind, visualising the impacts for service users and the pathways to change. In addition, it offers a practical resource for LEPs, helping them align delivery with the programme’s intended aims.

5.3.2 Refined Theory of Change (ToC)

The No One Left Behind ToC is structured around six main components. These are: (i) inputs, such as Scottish Government funding; (ii) setup, delivery, and monitoring activities, such as financial reporting templates; (iii) outputs, such as CV development support ; (iv) short-term outcomes, such as increased work readiness; (v) medium-term outcomes, such as increased participation in the labour market; and (vi) impacts, such as reduced unemployment rates.

The ToC describes these components and their interactions. Specifically, it presents how LEP and Scottish Government resources (i.e., inputs) and the actions and tools that support these initiatives (i.e., activities) shape short-term interventions (i.e., outputs), which in turn lead to short- and medium-term outcomes as well as longer-term impacts.

The ToC is structured around the three different funding strands within No One Left Behind. While recognising that service user groups may overlap (i.e. individuals can fall into more than one category), the ToC is set to reflect the distinct procurement routes for employability services, including targeted provision for parents and individuals requiring specialist employability support.

The following sections are structured around each of the aforementioned components. Please note that this section is best read alongside the visualisation of the refined ToC, available as a separate document.

Inputs and activities

Inputs for No One Left Behind reflect the “local by default, national by agreement” principle that underpins the Scottish Government’s approach to employability. The inputs are set out at two levels (national and local) and represent the resources initially committed to No One Left Behind-related work.

• At the national level, the Scottish Government provides funding (£90 million for the 2026-2027 period) for employability services through the Scottish Budget (Scottish Government, 2026). It also sets the strategic priorities and policies for these services, as outlined in the Employability Strategic Plan 2024-27 (Scottish Government, 2024), and establishes governance arrangements through a framework to inform the leadership and operation of LEPs (Scottish Government, 2025). In addition, a suite of resources/guidance documents has been developed to ensure a clear understanding of expectations, standards and commitments for employability services. These aim to deliver consistency while allowing local areas to develop and deliver services flexibly and in line with local needs and improve outcomes for employability participants. The No One Left Behind National Products Handbook (Scottish Government, 2025) brings these together and includes: (i) the Shared Measurement Framework, which guides data collection by identifying the questions that need to be answered; (ii) the Employability Service Standards, which set expectations for local delivery; and (iii) the Customer Charter , which outlines what service users can expect from these services and how to provide feedback. Other resources include the Service Design Toolkit; the Continuous Improvement Toolkit; Good Practice Guidance for Commissioning Employability Services in Scotland; No One Left Behind Data Products and the No One Left Behind Data Toolkit.
• At the local level,LEPsprovide local insight and use data and evidence to inform service provision, identifying the population groups with the greatest need in their area. They design and tailor services to individual and community needs, while drawing on frontline delivery skills and experience. On occasion, Local Authorities or other partners may also contribute financial resources

Activities for No One Left Behind represent the operational processes by which inputs are translated into specific interventions and how these are planned, delivered, and monitored, in chronological order. In more detail:

Set-up activities take place across both the national and local levels.

• At the national level,the Scottish Government is responsible forfunding, strategy and policy development, data analysis and evaluation . Example activities include allocating funding to Local Authorities in their role as Lead Accountable Body, managing relationships through dedicated relationship managers and overseeing delivery. Accountability is maintained through reporting requirements.
• At the local level,LEPs are the mechanism forcollaboration between Local Authorities (acting as the Lead Accountable Body), statutory partners (including the Department for Work and Pensions, Skills Development Scotland, the National Health Service, and higher education institutions), and the third sector. The Improvement Service also provides support to the functioning of LEPs. LEPs outline their delivery approach to the Scottish Government through Annual Investment Plans as part of the annual grant process. Local Authorities and LEPs manage local commissioning frameworks and are responsible for managing local commissioning and funding.

Delivery activities are carried out exclusively at the local level. In particular, LEPs deliver services by engaging communities, collaborating with providers, and involving service users in the design and co-creation of support tailored to local needs. LEPs provide flexible, person-centred support, including identifying specific groups as a priority, based on local need.

Monitoring activities take place across both the national and local levels:

At the national level, the Scottish Government standardises data and governance via the DRT, aligns policy across portfolios and pro duces quarterly statistics (Scottish Government, 2025).

At the local level, LEPs:

• Finance services and report back to the Scottish Government by submitting financial reporting returns.
• Monitor delivery, with Local Authorities as the Lead Accountable Body, sharing statistical programme returns and case studies with the Scottish Government, alongside tracking service provision for internal improvement.
• Embed lived experience in continuous learning by integrating co-design and service user feedback. This, in turn, informs future service delivery and contributes to continuous service improvement.

Funding strands

As previously mentioned, service provision is organised around the three core funding strands. As a result, the inputs and activities described above lead to specific outputs (interventions) aligned with their funding strand, while recognising some overlap. While LEPs have significant autonomy in how they design and deliver services, they are also required to demonstrate how they will reach and support three core groups^[4], each associated with a specific funding strand:

• All-Age Employability Service covers the provision of flexible, person-centred support for people aged 16-64, including targeted support for specific groups identified as local priorities. It accounts for £39 million of the £90 million total allocated for employability services in 2026–27.
• Child Poverty Funding (Parental Employability Support) focuses on supporting parents, including specific parental groups identified as local priorities. It accounts for £40 million of the £90 million total allocated for employability services in 2026–27.
• Specialist Employability Support (SES) is aimed at disabled people and people with long-term health conditions who require specialist support. It represents a smaller share (£5m) of the total employability services budget.

Outputs-interventions

Outputs reflect the services and interventions procured under the three funding strands. LEPs commission locally planned, coordinated, and co-produced front-line services and undertake targeted outreach to engage service users. They also provide key worker support, structured around the five-stage employability pathway model, namely (i) referrals, engagement and assessment; (ii) needs assessment; (iii) vocational activity; (iv) employer engagement and job matching; and (v) in-work support and aftercare.

Alongside the key worker support model, LEPs provide a range of services. These can be delivered as part of the employability pathway model or on a more targeted or ad hoc basis. Services are aligned to the following categories:

• Confidence and essential skills development, including one-to-one or group sessions, mental health and wellbeing support activities, and transferable skills training.
• Qualification pathways, including access to universities/further education, accreditation and qualification routes (apprenticeships, work-based qualifications and volunteering qualifications), sector-specific training, and digital skills certifications.
• Job search support, including career information advice and guidance, CV development assistance, and self-employment support.
• Job matching, including employer referrals and direct job placements.
• Income maximisation support, including advice and guidance on benefits, as well as support to better manage personal finances.
• Parental childcare and barrier-reduction support, including assistance to identify and source childcare.
• Parental employment, in-work progression and sustainment, including supporting individuals into jobs that offer flexible employment practices (such as hybrid working, part-time roles and school-friendly hours), and gradual entry models (e.g. low-pressure environments) to support those long-term unemployed.
• Specialist support, including tailored services for people with disabilities and health conditions, people with learning disabilities and autistic people, ex-offenders, people experiencing homelessness, individuals recovering from addiction, care-experienced people, refugees, and migrant workers.

Outcomes

The intermediate results from the outputs represent outcomes, which are grouped chronologically into short- and medium-term outcomes. Within each time horizon, they are also organised thematically by beneficiary group and type of anticipated benefit.

Short-term outcomes include both improvements to the delivery of employability interventions and results for service users.

Employability intervention delivery improvements include (i) improved partnerships between Local Authorities and involved members (e.g. service providers, etc.); (ii) services that are easier to access and navigate for users; (iii) reduced duplication/better coordination of services; and (iv) a “no-wrong-door” approach ensuring people are guided to the right support at the right time.

For service users, short-term outcomes relate to:

• Work readiness, covering increased confidence/motivation to seek and sustain employment, improved soft skills, and development of skills aligned with labour market needs and personal aspirations.
• Continuing learning, spanning accredited qualifications, training or internships, and access to further or higher education.
• Employment outcomes relate to greater awareness of job opportunities, access to appropriate job opportunities, successful job placements, increased career progression opportunities, improved job retention, and higher participation in volunteering.
• Financial management, including increased awareness of eligible benefits and better personal money management.
• Parent-specific outcomes encompass increased access to employment through improved childcare support, enhanced ability to balance work and family responsibilities, greater confidence in managing parenting alongside career goals, and improved household financial stability.
• Disability-related outcomes relate to improved employment readiness for disabled people, increased access to suitable jobs accommodating disabilities, and greater confidence to pursue employment.

Medium-term outcomes are the expected results of service provision that have a longer materialisation timeframe than short-term outcomes. These outcomes are expected across the following areas:

• Employment: Indicative examples include increased participation in the labour market across service users, including priority groups; more individuals progress into sustainable, fair, and better-paid employment aligned with their circumstances; increased job satisfaction and workplace confidence.
• Income maximisation, consisting of higher income through (i) access to employment; (ii) better-paid roles and qualifications; and (iii) claims of eligible benefits.
• Household outcomes: These include improved household financial resilience; better management of work and family responsibilities; increased confidence and self-efficacy as a parent; enhanced ability to plan longer-term household decisions; greater engagement with health, education, or support services for the family.
• Equitable employment: increased financial resilience through sustained or better-suited employment; greater ability to balance work and caring responsibilities, particularly for disabled people; enhanced independence and career aspirations.
• Health and wellbeing: enhanced self-esteem; reduced stress; improved management of healthcare needs; stronger sense of personal achievement and self-efficacy.

Impacts

Impacts represent the long-term changes expected to occur and are aligned with the strategic goals of No One Left Behind. These reflect system-wide and societal changes, namely:

• A more effective employability support system, with improved access to and experience of services
• Reduced unemployment rates
• Improved workforce skills aligned with economic demand
• A strengthened local economy
• Reduced reliance on unemployment benefits^[5]
• Reduced child poverty rates
• A reduced disability employment gap
• Reduced demand for social and support services
• Healthier populations
• Stronger community cohesion and reduced antisocial behaviour. Evaluation Considerations

5.4 Evaluation considerations

This chapter presents our findings regarding (i) possible counterfactual scenarios and control groups for the No One Left Behind evaluation; (ii) feasible impact evaluation approaches; and (iii) how the outcomes identified in the Impact Evaluation can be monetised to determine the value for money of No One Left Behind.

5.4.1 Counterfactual scenario & control groups

The counterfactual represents a theoretical scenario of what would have occurred if a policy had not been implemented. It serves as the benchmark against which the actual outcomes of an intervention are measured to determine the extent to which the policy itself was responsible for any changes. A control group is the specific group of non-participants used to practically measure the counterfactual.

As mentioned in the Methodology section, our desk-based review and stakeholder interviews have allowed us to examine several potential control groups. This section provides a more detailed assessment of these options, considering both their rationale and practical limitations.

5.4.2 Overarching considerations

There are several overarching considerations that apply to all potential control groups presented below. In particular:

(i) the robustness of the analysis depends heavily on the available sample size - larger and more representative samples improve the reliability and statistical power of findings;

(ii) access to individuals’ past labour market history is key, as it allows tracking and comparisons of employment trajectories over time for both the treatment and control groups;

(iii) there is no single dataset capturing the full range of outcomes of interest; thus, all control group options require combining multiple data sources;

(iv) to identify individuals comparable to No One Left Behind participants who did not receive support, detailed background information is needed to ensure that the control and treatment groups differ only in terms of their No One Left Behind participation status. The key variables on which the estimation of weights would be based include:

• Demographic characteristics (e.g. age, sex, geographic location) in RAPID.
• Employment history (prior employment spells and earnings trajectories) in RAPID/RTI.
• Benefits history (type, duration and intensity of benefit receipt) in RAPID.
• Health-related indicators, where feasible. These are available as proxies in RAPID (through receipt of health-related benefits, such as Employment and Support Allowance (ESA) and Personal Independence Payment (PIP)), though these capture only the subset of individuals who have successfully applied for and been awarded these benefits.

It is also worth noting that the weighting approach can be used to control for local characteristics and variability in service provision across LEPs. This can be achieved through a set of “local area control” variables, such as local unemployment rates, wages, sectoral distribution, urbanisation levels, and demographic characteristics. Together, these variables constitute a statistical description of specific areas or groups of areas that can then be used as weights in the inverse weighting process.

5.4.3 Possible control groups

Below, we present the potential control groups and assess their feasibility. Control groups are listed in order of suitability for the national impact and economic evaluation of No One Left Behind.

In summary, we believe “unreached non-participants” is the most suitable option, followed by “participants funded by UK and other SG programmes”. Options (3) and (4) are not feasible in practice.

A. Unreached non-participants: This group includes individuals who share similar observable characteristics with No One Left Behind participants and who could potentially benefit from No One Left Behind support, but who have not engaged with the approach.

In principle, this would represent the most appropriate comparison group, as it closely reflects the intended target population. The counterfactual scenario is that in the absence of No One Left Behind, participants would not have been identified or reached by the approach and would therefore have received no support.

However, in practice, there is a risk of selection bias. Individuals who are not reached may differ in important but unobserved ways (such as motivation or personal circumstances), which could influence both their likelihood of participating and their outcomes.

B. Participants funded by UK and other SG programmes: This group includes individuals who have previously received employability support through programmes other than No One Left Behind, such as Scotland’s Employer Recruitment Incentive and Jobcentre Plus.

Conceptually, this is a plausible comparison group, as these individuals are also receiving employment support. In this case, the comparison reflects a scenario where similar individuals access an alternative provision rather than No One Left Behind.

However, there are important practical limitations. In reality, individuals may receive support from multiple programmes simultaneously, and Local Authorities may combine No One Left Behind funding with other funding streams. This overlap makes it difficult to isolate the specific impact of No One Left Behind, as outcomes cannot be clearly attributed to a single programme or approach.

C. Drop-outs (referred to as non-starters): This group includes individuals who were referred to No One Left Behind but chose not to engage, as well as those who initially engaged but later disengaged from the service. In this case, the counterfactual scenario is that No One Left Behind participants would have chosen not to engage with the services and would have received no support.

While this group may offer a natural comparison, there are notable limitations. Individuals who disengage or do not take up support may differ systematically and substantially from participants in unobserved variables, such as motivation levels, personal barriers, or readiness for employment. These differences can bias impact estimates.

Furthermore, the concepts of “drop-out” or “non-starter” are less clearly defined in the No One Left Behind context. Due to the approach's flexible, person-centred nature, individuals may vary their level of engagement over time as their needs change. As such, disengagement is not always permanent or clearly identifiable, making it challenging to define and use this group consistently for evaluation purposes.

D. Fair Start Scotland (FSS): This group includes people who received employability support through FSS, the previous national model in Scotland. In theory, this would be a robust comparator that would allow the Scottish Government to assess which of the two models is more effective. The counterfactual in this case is that No One Left Behind participants would have received FSS support instead.

However, there are substantial differences in eligibility criteria and how FSS was funded and delivered compared to No One Left Behind. These differences in design and funding structures imply that the two are not directly comparable. Stakeholders in interviews also raised concerns that any comparison between participants in the two groups may not yield a reliable estimate of the impact.

5.4.4 Impact evaluation

There are several alternative approaches available for impact evaluation. Based on HM Treasury’s Magenta Book, they can be broadly categorised into (i) experimental methods; (ii) quasi-experimental methods (QED); and (iii) theory-based methods. Below, we explain in detail the possible approaches in each category and examine their theoretical robustness and practical feasibility.

Experimental methods

Experimental designs are considered the most robust means of establishing attribution and measuring the size of an effect. The defining characteristic of these methods is the use of random allocation to assign individuals to the treatment and control groups. This process ensures that there are no systematic differences, whether observable or unobservable, between the groups at the start of the study, meaning any subsequent divergence in outcomes can be confidently attributed to the intervention itself.

The most common experimental method is the Randomised Controlled Trial (RCT). Individual-level randomisation assigns specific individuals to groups, whereas cluster-randomised trials assign entire groups (such as Local Authorities) to conditions. In instances where it is ethically difficult to withhold a service, a waiting list design can be utilised. In this case, individuals are randomly assigned to either immediate treatment or a waitlist, allowing for a short-term comparison against a counterfactual before the control group eventually receives the intervention.

Despite their rigour, experimental methods require careful execution and sufficient sample sizes to ensure statistical power, and they must be integrated into the programme design well before implementation begins. Finally, the assignment of individuals between the two groups must be random.

Conclusion: Experimental approaches are not feasible for No One Left Behind, as they must be built into the programme's design. They cannot be applied retrospectively as they affect who gets access to services and who does not.

Quasi-experimental methods

In many policy contexts, random allocation is unfeasible due to legal mandates, ethical considerations, or the practicalities of national rollouts. In such cases, quasi-experimental methods provide a high-quality alternative by using statistical techniques to approximate experimental conditions. We explore the feasibility of all common QED approaches below, following HM Treasury Magenta Book guidance.

Regression discontinuity (RD) is used to measure impact when individuals are assigned to a programme based on a clear cutoff point (e.g. income level). The rationale is that people just above and just below the cut-off are likely to be nearly identical, except that one group receives the treatment and the other does not.

Conclusion: For the evaluation of No One Left Behind, an RD is not suitable because the delivery of No One Left Behind services does not involve a clearly defined eligibility threshold. Simply put, there is no specific cut-off point or rule (such as an income level, education, or age) that strictly determines who receives support and who does not. In the case of No One Left Behind, access to services is typically based on a more flexible assessment of individual needs rather than on fixed, rule-based criteria.

Propensity score matching (PSM) involves pairing participants who received the intervention with non-participants based on observable characteristics that determine their likelihood of participation. This creates a comparison group that, while not randomly selected, is statistically similar to the treatment group across all known relevant variables.

In practice, a statistical model (typically logistic regression) could be first used to estimate each individual’s probability of participating in the programme based on observed characteristics (the propensity score). Treated individuals are then matched to one or more non-participants with similar propensity scores. Once matching is complete, outcomes for the treated group and the matched comparison group can be compared directly. The difference in average outcomes between these groups could be used to estimate the average treatment effect on the treated (ATT).

The validity of PSM depends on two key conditions:

• Matching must be performed using characteristics that are either fixed (e.g. gender, year of birth) or collected before the intervention to ensure the treatment itself has not altered those traits.
• Assignment to treatment depends only on observable characteristics. If unobservable factors (such as individual motivation) affect both participation and outcomes, the impact estimate will be biased.

Conclusion: A PSM approach is theoretically feasible, subject to sufficient data availability (see the Evaluation Recommendation section).

Similarly, the difference-in-differences (DiD) approach measures impact by comparing the changes in outcomes between two groups over time. Instead of direct matching, it compares the change in outcomes for the treated group before and after the intervention with the corresponding change for a non-treated comparison group.

In practice, this is implemented by observing outcomes for both groups over at least two time periods (pre- and post-intervention), calculating the change over time for each group, and then subtracting the change in the control group from the change in the treated group. This “difference-in-differences” estimator isolates the effect of the intervention under the assumption of parallel trends. If the treatment and comparison groups' historical trends moved in parallel, the comparison group’s trajectory after implementation can serve as a credible counterfactual for the treated group.

The key assumptions underpinning DiD are:

• The fundamental condition is that outcomes for both groups must have historically moved in parallel. This means that in the absence of the intervention, the treatment group's trend would have continued to follow the comparison group's trajectory.
• The DiD method also relies on the assumption that no other external factors or contemporaneous interventions occurred that affected one group but not the other during the evaluation period.

Conclusion: A DiD approach is theoretically viable, conditional on identifying a control group that satisfies both conditions (see the Counterfactual section for more details).

Other, less relevant QED methods include:

• Synthetic control method (SCM), which constructs a "clone" of an intervention area by using a weighted average of other non-treated areas to approximate the treated unit’s historical performance. This is primarily a “macro-level” approach, meaning it does not account for impacts and variations on the individual level. As a result, it is less relevant to No One Left Behind, where HMRC RTI and the DRT provide access to individual-level data.
• Interrupted time series (ITS) analysis, or before-and-after analysis, as it is most commonly known, involves collecting numerous data points before and after a time-marked intervention. Evaluators look for an "interruption" in the pre-existing trend - such as a sudden change in level or a change in the slope of the trend line - and attribute that deviation to the intervention itself. A key requirement for this approach is a clear intervention point, which, in the case of No One Left Behind, does not apply because there was a phased rollout. Furthermore, the rollout of No One Left Behind in 2019 coincided with significant global changes (e.g., the COVID pandemic), which hinder attributing any changes in trends before and after 2019 to No One Left Behind alone.
• Instrumental variables (IV), which identify an external factor (the instrument) that influences participation in a programme but has no direct effect on the final outcome, thereby isolating the policy’s impact from self-selection bias. However, the IV approach can identify the local average treatment effect (LATE), meaning it only estimates the impact on those who were "on the margin" of participating because of the instrument, rather than the average effect for the entire population. This limits its usefulness for evaluating No One Left Behind, as there is no evidence that the majority of the treated population would be on the participation margin. Furthermore, finding a variable that affects participation in No One Left Behind but not the outcomes (to be used as the “instrument”) would be challenging.
• Timing of events (or duration modelling) is an approach occasionally used in labour market evaluations. It involves modelling the amount of time which elapses before a given event, or the time an individual spends in a given state, for example, how long somebody remains unemployed. For example, in an intervention aimed at helping benefit claimants return to work, this would be the time at which they participate in the intervention, and the duration thereafter before they leave benefits. In the case of No One Left Behind, the concepts of “drop-outs” and “re-engagement” cannot be defined robustly due to the programme's flexibility. As a result, the duration of engagement cannot be consistently defined, rendering the timing-of-events approach infeasible.

In conclusion, by combining elements from the two most promising methods, namely PSM and DiD, we can create a “doubly robust” DiD framework, following Sant’Anna and Zhao (2020). This combines two components

• Inverse probability weighting (IPW): Weights are constructed based on estimated propensity scores (the probability of being in the treatment group given observed characteristics), so that the weighted treatment and comparison groups are comparable at baseline on observed characteristics. Unlike one-to-one matching, which discards unmatched observations, weighting uses all observations in the sample.
• DiD estimation: Outcomes are compared between treatment and comparison groups before and after programme entry, with the DiD accounting for time-invariant unobserved differences between groups.

The “doubly robust” property means that the estimator produces consistent impact estimates if either the propensity score model or the outcome model is correctly specified. This means that only one of the two needs to be wrong for the estimator to recover the causal effect. Outcome variables would be measured using consistent definitions across groups, primarily using RAPID/RTI.

The approach allows for explicit testing of pre-treatment trends and provides a flexible framework for handling differences in the timing of programme entry. It is consistent with best practice in quasi-experimental evaluation and is expected to achieve a rating of at least SMS Level 3, subject to data quality and the strength of the weighting.

5.4.5 Theory-based methods

While experimental designs focus on the size and significance of change, theory-based methods assess whether an intervention had an impact by exploring how and why it occurred, the influence of context, and the extent to which the results are generalisable. It is important to note that while these methods allow for the attribution of causality, they do not provide the precise statistical estimates of effect sizes.

Theory-based methods do not require quantitative data and are thus often used to estimate the impact on outcomes that are hard to quantify, or where there are significant data gaps.

To get a better understanding of which theory-based method would be best suited to conduct the evaluation, we assessed the feasibility of various options, detailed below.

• Contribution analysis seeks to establish a credible "contribution claim" by demonstrating that an intervention's activities were implemented as intended and that the evidenced chain of results aligns with an established ToC. Typically, an intervention contributes to changes at different stages, such as short-term outcomes (e.g., work readiness), medium-term outcomes (e.g., employment status), and eventual impact (e.g., a more effective employability support system). The contribution of the intervention is assessed at all these different stages, creating a ‘contribution story’ that outlines how and why change has (or has not) happened. Conclusion: Contribution analysis is feasible for No One Left Behind. The ToC framework established and validated as part of this project would significantly expedite contribution analysis, minimising the resources and time required. The main challenge would be to ensure adequate stakeholder participation across areas and No One Left Behind services to establish the contribution claims.
• Realist evaluation focuses on understanding the mechanisms through which an intervention could impact the outcomes of interest, seeking to answer the question ‘what works, for whom, in what circumstances?’. This method operates on the core formula that ‘context + mechanism = outcome’, exploring what mechanisms are likely to operate in different contexts and the outcomes that will be observed when they do. These hypotheses are then evaluated and refined using available data. Conclusion: This method is also feasible in the case of No One Left Behind. In practice, it would seek to establish the conditions (or context) under which the outputs and outcomes of the refined ToC are achieved. While it provides useful information on the conditionality of impacts, it is often very resource-intensive as it requires in-depth engagement with stakeholders.
• Most significant change (MSC) involves analysing stories of change and deciding which account is the most significant. This method involves significant engagement with stakeholders, discussing and recording stories related to key domains of change to reflect the programme's or intervention's key impacts. For example, one participant may share how No One Left Behind support helped them secure internships that led to a permanent job. Another participant may describe how No One Left Behind helped them gain confidence to re-enter education after a long period of unemployment. Then, through stakeholder discussions, a story such as securing permanent employment post-No One Left Behind support may be selected as the most significant change if it is considered particularly policy-relevant. Conclusion: MSC is feasible in theory, although it does not capture all impacts, requires significant stakeholder engagement, and is often very resource-intensive and time-consuming. MSC is by far the most costly option, as it would require 1-1 sessions with multiple stakeholders, including various types, to validate the stories and determine which is most significant.

5.4.6 Estimation of outcomes

The aim of the impact evaluation will be to assess whether No One Left Behind has been effective in achieving its intended outcomes. To that end, our feasibility assessment explored which short- and medium-term outcomes, identified through the ToC, can be measured and how. For the purposes of both the impact and economic assessment of No One Left Behind, the evaluation is broadly centred around four key outcome categories: (i) employment, (ii) education, (iii) health, and (iv) wellbeing. These aspects capture the core areas where the programme is expected to generate meaningful improvements. The system-wide operational outcomes identified in the ToC should be explored as part of a standalone process evaluation.

It is important to recognise that the nature of the four main outcomes varies, and this has implications for how they can be measured and evaluated. Some outcomes are supported by clearly defined indicators and are captured within existing quantitative datasets. These typically include measurable aspects such as employment status, earnings, or participation in education and training.

However, other outcomes are more qualitative in nature and relate to factors such as individual experiences, confidence, motivation, and overall wellbeing. These outcomes are not directly observable through administrative data. As a result, qualitative methods such as stakeholder interviews, participant feedback, and case studies would be required to complement the quantitative analysis and provide a more comprehensive picture of the programme's effects.

The table below summarises the coverage of the respective outcomes in the refined short list of data sources. Overall, where possible, the same dataset should be used for both the control and the treatment group.

Outcome	Control group	Treatment group
Employment	Coverage of employment status, earnings, and other labour market outcomes for comparable individuals not participating in No One Left Behind is extensive and is primarily available in HMRC RTI. However, a new data access agreement would need to be established.	For No One Left Behind participants, employment outcomes are recorded in the DRT. To address data quality concerns and ensure consistency, No One Left Behind participants should be identified in the HMRC RTI dataset by using their NI number.
Education	Education-related outcomes are included in LFS/APS surveys. However, it is worth noting that these are surveys, so they do not capture the entire population, and their sample sizes for Scotland are small. Furthermore, they do not include NI numbers, hindering data linkage and posing a risk of contamination to the control group (i.e., No One Left Behind participants cannot be reliably excluded).	For No One Left Behind participants, education outcomes are recorded in the DRT, noting the relevant data quality concerns. While the LFS/APS includes data on No One Left Behind participants, the participants cannot be identified without an NI number.
Health	While some health-related outcomes are captured by LFS/APS surveys, the list of variables and the sample size are limited. Health-related data is also captured by NHS Scotland. However, the relevant datasets use CHI numbers rather than NI numbers, which hinders data linkage to the DRT. Even if a linkage were established, there would likely be concerns about sharing individuals' sensitive healthcare data.	For the treatment group, some health-related information is captured, but it captures almost exclusively long-term conditions and disabilities. These conditions are unlikely to be affected by No One Left Behind, at least not in the short- to medium-term.
Wellbeing	The Scottish Health Survey includes data on wellbeing, mental health, and loneliness. While this is a survey, it is Scotland-specific and thus has better coverage of the Scottish population.	Wellbeing data is not routinely reported for No One Left Behind participants. Assessing No One Left Behind’s impact on this outcome would likely require new primary data collection through surveys, focus groups, or interviews.

5.5 Economic evaluation

This section presents our suggested approaches for conducting an economic evaluation of No One Left Behind, compared with the selected control group and the counterfactual mentioned earlier. While impact evaluation focuses on establishing causality and attribution, economic evaluation is primarily concerned with answering questions regarding the net value and whether a programme represents value for money for the Exchequer or the wider society.

5.5.1 General considerations

According to the HMT Green Book, there are two methodologies for conducting economic evaluations:

• Social Cost-Benefit Analysis (CBA) seeks to assess the return to society as a whole in monetary figures, assigning a “£” figure in both costs and benefits. It looks beyond the public sector, accounting for societal and environmental impacts. Its key outputs are the Net Present Social Value (NPSV) and the Benefit-Cost Ratio (BCR).
• Cost-Effectiveness Analysis (CEA) is an alternative economic evaluation method, used when benefits are hard to monetise. CEA involves estimating the social costs in monetary terms but measuring the benefits in quantifiable, non-monetary units of effectiveness, such as "lives saved," "cases detected," or "hectares of land restored". CEA results are reported in (i) the Cost-Effectiveness Ratio (CE), which measures the cost of delivering a single unit of outcome; and (ii) Net Present Unit Cost (NPUC), expressing the quantifiable benefits relative to their real discounted social cost.

For the economic evaluation of No One Left Behind, the most robust option would be a CBA. This is because most outcomes can be monetised, No One Left Behind is expected to have significant societal benefits that should be accounted for, and the BCR and NPSV are key metrics for public policy appraisal. As a result, the rest of this section presents how an SCBA for No One Left Behind could be undertaken.

5.5.2 Social Cost-Benefit Analysis

This method systematically compares the total costs of delivering the programme with the total benefits it produces, allowing for an assessment of whether the intervention represents good VfM. The CBA could be conducted from two complementary perspectives: the Exchequer (or fiscal impacts) and society as a whole.

From the Exchequer perspective, the evaluation will focus on direct fiscal impacts on the public sector. This includes changes in (i) tax revenues resulting from increased employment and earnings; and (ii) government expenditure, such as welfare benefit payments. The fiscal BCR resulting from this process is a key metric for policymakers, as it allows for a direct assessment of the programme's implications for public finances. However, it represents a relatively narrow view and does not capture broader impacts, such as increased productivity across the wider economy.

The societal perspective considers wider economic and social outcomes. These include increases in economic output (proxied by individuals’ earnings), improvements in health and wellbeing, and other benefits that may not directly affect government budgets but contribute to overall social value. However, from a societal perspective, some impacts, such as taxes and benefit payments, are treated as economic transfers. As a result, they largely offset each other, unless there are significant redistribution effects.^[6]

By considering both perspectives, the economic evaluation provides a more comprehensive assessment of the programme’s overall impact. Below, we present our approach to monetising No One Left Behind’s costs and benefits, while highlighting differences between the fiscal and societal perspectives where applicable.

5.5.3 Benefits

As outlined in the Impact Evaluation section, the main benefits of No One Left Behind are: (i) employment; (ii) earnings; (iii) education; (iv) health; and (v) wellbeing. We discuss each of these in turn in the following subsections, focusing on individual-level impacts. These can then be aggregated by multiplying by the number of No One Left Behind participants affected by each impact.

Employment

In an SCBA, it is sometimes assumed that a job filled by one person is simply a job vacated by another elsewhere in the economy (i.e. the principle of displacement). However, for an employability intervention such as No One Left Behind, this principle does not hold, as individuals would otherwise remain unemployed or economically inactive.

As a result, there will be the following monetisable impacts: (i) a reduction in social security payments; (ii) an increase in taxes paid; and (iii) an increase in economic output. The first two impacts are considered an economic transfer from a societal perspective, while the third is covered in the Earnings subsection. This means that the amount lost by one person in taxes paid (or welfare benefits not received) is offset by an equal gain for someone else in society, who receives other benefits financed by those taxes. To the extent that the socioeconomic status of those paying taxes and those receiving benefits does not differ substantially, the CBA methodology suggests that the two effects cancel out on the societal level (i.e. there is no redistribution effect).

From a public-sector perspective, moving an individual from unemployment to work is a significant financial benefit. Higher levels of employment lead to increased tax revenues (through income tax and national insurance contributions) and reduce social security expenditure. Both impacts are realised benefits for the Exchequer, which is concerned only with financial inflows and outflows from the Treasury.

In practice, the increase in tax revenue can be calculated directly through the specification of the regression by adding the tax formula as another variable. Reductions in social security payments can be observed directly for each individual if data linkage is established with RAPID. Otherwise, assumptions will need to be made about the average reduction in benefit claims per £1 of earnings.

Fiscal savings from reduced reliance on welfare benefits can be estimated using established benchmarks from the literature. For example, a study by the Public Service Transformation Network and New Economy estimates that the annual monetised fiscal benefit of one additional individual entering employment is approximately £9,234^[7].

Earnings

From a SCBA perspective, wage increases are considered a transfer because they represent a cost to the employer and an equal benefit to the employee. Focusing on financial flows, it is a zero-sum movement of money from one party to another. However, under standard Green Book assumptions of competitive labour markets, wages can be used as a proxy for the marginal productivity of labour. In this context, changes in earnings can be interpreted as reflecting changes in economic output attributable to the intervention.

In practice, the increase in output due to higher earnings can be calculated as the difference in earnings between No One Left Behind participants and the control group, as per the selected impact method and counterfactual control group (see the respective sections).

Education

At the individual level, No One Left Behind is likely to increase participation in higher education and training.^[8] As individuals gain additional qualifications, skills, and knowledge, this strengthens their employability and improves their chances of accessing more stable, higher-paying jobs, thereby increasing their lifetime earnings and economic output. The impacts of higher education are expected to be realised in the longer-term. This is because individuals typically need time to complete their education, enter the labour market, and gain experience before the full gains are observed.

However, it is worth noting that there is no robust approach to estimate the impact of No One Left Behind on education through a QED approach. The main source of information would be the LFS/APS surveys, which do not use NI numbers, meaning they cannot be linked to the DRT and HMRC RTI. This, in turn, prevents the identification of No One Left Behind participants and their exclusion from the control group. To address this issue, the evaluator can construct a control group of comparable individuals in the APS/LFS from other UK nations, excluding Scotland, and thus ensuring the control group does not include No One Left Behind participants.

Another concern is that other UK nations have introduced or scaled up their own employability programmes, which may contaminate the control group’s outcomes. This issue can be partially mitigated by comparing education outcomes for earlier years, exploiting the fact that No One Left Behind was introduced in 2019, when many UK employability programmes were not yet fully rolled out. Finally, the LFS/APS would not reflect any long-term education programmes, as it only tracks individuals for up to 12 months. All things considered, education is likely challenging to evaluate through QED approaches and would be better suited to a theory-based approach supported by quantitative evidence.

The monetisation of any identified education outcomes depends on the specific services and the type of education achieved. If No One Left Behind services facilitate progression to further and higher education, then there may be an increase in lifetime earnings. To monetise this increase, the evaluator should compare educational outcomes between the treatment and control groups and then use estimates of the impact of education on earnings from existing research. For example, a report by IFS estimates that the discounted lifetime increases in gross earnings between those who attend and those who do not attend higher education are £240k for men and £140k for women. This is after accounting for the fact that not all people who receive higher education find employment.

As mentioned earlier, the increase in earnings is considered a transfer in societal terms. However, it serves as a proxy for increased employee output and is therefore included in an SCBA model. From a fiscal perspective, the impacts of education are mediated through increased life-time earnings, so the same rationale applies, as detailed in the Earnings section.

Health

No One Left Behind is likely to contribute to improved health outcomes through its effects on employment, income, and access to higher education. As individuals get employed and their earnings increase, they will likely gain better access to healthcare services.

From a fiscal perspective, improved population health can reduce pressure on public health services by reducing hospital bed stays, shortening waiting times, and service-related costs (e.g. fewer A&E incidents and ambulance calls). Annual savings to the NHS from improved health can be estimated using established benchmarks from existing research. For example, a study by the Public Service Transformation Network and New Economy estimates that the annual monetised savings to NHS England due to improved health for one additional individual entering employment are approximately £566^[9].

For society, a commonly accepted approach is to assess changes in health using quality-adjusted life years (QALYs), a standard measure that captures both the quality and quantity of life. Quantity of life refers to the number of years a person lives, or, more specifically, the additional effective years of life gained due to an intervention. Quality of life, on the other hand, reflects the health-related wellbeing experienced during those years, indicating how good or poor a person’s health is over time. Analysis by the Department for Work & Pensions (DWP) suggests that an individual returning to employment gains 0.068 QALYs. This can then be monetised using the HM Treasury-recommended estimate of £60,000 per QALY (in 2014 prices).

Wellbeing

No One Left Behind is also likely to have a positive impact on individual wellbeing. Apart from dedicated wellbeing and mental health offerings, employment and education are associated with improved mental health on their own (e.g. research by SIPHER).

These wellbeing benefits are not included in the fiscal analysis because they do not affect Treasury financial flows. In the SCBA, a common approach to estimating wellbeing effects is using well-being-adjusted life years (WELLBYs). A WELLBY is defined as a one-point change in life satisfaction (on the 0–10 scale) experienced by one person for one year. The changes in life satisfaction that have been estimated and expressed in WELLBY terms can then be converted into monetary values. This is done by multiplying the total WELLBYs by the recommended valuation of £13,000 per WELLBY (in 2019 prices and values). However, the WELLBY and QALY approaches should not be combined, as there is a significant risk of double counting.

Finally, it is worth noting that quantifying the impact of No One Left Behind on mental health and wellbeing is challenging, as described in the Impact Evaluation section. Most likely, this benefit will be estimated using theory-based approaches, hence rendering its monetisation challenging and mostly assumption-driven.

The table below summarises the aforementioned benefits and monetisation suggestions.

Impact	Fiscal perspective	Societal perspective	Monetisation approach
Employment & earnings[10]	i) Reduction in social security payments ii) Increase in tax contributions	i) Increase in economic output	i) The increase in tax revenue can be calculated directly through the specification of the regression, by adding the tax formula as another variable. ii) If data linkage is established with RAPID, the impact on social security payments can be observed directly for each individual. Otherwise, assumptions will need to be made about the average reduction in benefit claims per £1 of earnings.
Education	i) Increase in taxes due to an increase in lifetime earnings from improved education	i) Increase in economic output due to higher lifetime earnings from better education	i) Due to data gaps, education impacts will most likely be examined through theory-based approaches. Any quantitative estimates can be monetised depending on the type of services offered and the type of education achieved. If there are any substantiated impacts on further and higher education, these can be monetised by their effect on lifetime earnings.
Health & wellbeing	i) Reduced NHS costs due to better health and wellbeing ii) Improved physical health	i) Wellbeing benefits	i) Public savings on NHS can be calculated using NHS Scotland unit costs for different services, or directly through estimates such as a study by the Public Service Transformation Network and New Economy. ii) Improvements in physical health for the society can be estimated using QALYs for the respective health improvement and then monetised through the valuation estimate from the Green Book. iii) Wellbeing impacts can be quantified using WELLBYs or SIPHER estimates, which are then monetised using Green Book estimates.

5.5.4 Costs

To assess the overall VfM of the No One Left Behind programme, the analysis will compare total benefits and total costs to calculate the net economic impact. Several key data sources can support the identification and measurement of programme costs. Based on initial scoping work and stakeholder discussions, two primary sources stand out:

• Annual Investment Plans: AIPs outline planned spending and priorities across different strands of the No One Left Behind programme for each financial year. These documents provide a clear overview of intended resource allocation and strategic focus.

• Quarterly Finance Reports: These offer more detailed information, including breakdowns of expenditure across categories such as staffing, participant-related costs, and operational expenses.

However, one important challenge in accurately estimating programme costs emerged during stakeholder interviews. Stakeholders highlighted that LEP employability services are funded through multiple streams, making it difficult to identify which part of the achieved outcomes stems from No One Left Behind funding. This is compounded by the use of broad expenditure categories (e.g. internal staffing, delivery, commissioning, training, and incentives), which are interpreted inconsistently by Local Authorities. As a result, data is not fully comparable across areas.

5.5.5 Overarching considerations for CBA

There are several overarching factors to consider when designing and conducting an economic evaluation. Below, we present a summary of the most important ones.

Deadweight

Both costs and benefits that arise specifically because of No One Left Behind and would not have occurred otherwise. Any outcome that would have happened anyway in the Business-As-Usual (BAU) scenario is considered deadweight and must be excluded from the approach’s reported impact. For example, if No One Left Behind helps an individual enter employment, they would have found, anyway, through standard market trends, that "gain" is not included in the CBA.

Deadweight is closely linked to the counterfactual and control groups detailed in the impact evaluation chapter. As a result, deadweight would be accounted for by estimating the “net” impacts over and above those of the selected control group. For example, if the counterfactual compares No One Left Behind to another programme such as FSS, the analysis should include only the incremental costs of No One Left Behind beyond what FSS would have required.

Optimism bias

Optimism bias is defined as the "demonstrated systematic tendency" for evaluators to be over-optimistic about key assumptions in an appraisal. This bias typically manifests in three specific ways: (i) capital and operating costs are higher than anticipated in reality; (ii) social benefits are lower than expected; and (iii) the time taken to deliver the intervention is longer than originally forecast.

To ensure more accurate estimates, the Green Book suggests accounting for optimism bias by making explicit adjustments at the outset. This involves increasing estimated social costs and project durations and decreasing estimated social benefits based on an adjustment factor. The size of the percentage adjustment should be informed by the evaluator's own evidence of "average historical forecast errors. Alternatively, the Green Book suggests applying upper- and lower-bound adjustment factors ranging from 0% to 41% for outsourced projects.

Social discounting

Social discounting is a method used to compare costs and benefits occurring at different times consistently. It translates future values into "present value" terms, allowing practitioners to aggregate impacts that occur over the entire lifetime of a proposal. Discounting is based on the principle of social time preference, which acknowledges that people generally prefer to receive benefits sooner and incur costs later. Consequently, impacts occurring further in the future are given less weight than those occurring sooner. Depending on the type of impact, there are different discount rates:

• The social time preference rate (STPR): This is the real discount rate used in government appraisals. For the first 30 years of a proposal, the standard rate is 3.5%. This rate declines for longer-term impacts: 3.0% for years 31–75, and 2.5% for years 76 onwards. All identified impacts, apart from health and wellbeing, should be discounted at these rates.

• Health and life discount rate: The identified health and wellbeing impacts (QALYs) are discounted at a lower, constant rate of 1.5%. This lower rate reflects the fact that the utility derived from additional years of life does not decline even as real incomes rise.

Sensitivity & stress-testing

Given evaluations rely on assumptions that may not materialise, sensitivity analysis is used to test the robustness of the results. This process involves testing how changes in key assumptions, such as input costs, service demand, or the size of social benefits, affect the proposal's summary metrics (e.g., the Benefit-Cost Ratio). Stress-testing involves examining specific scenarios rather than arbitrarily varying parameters. This might include simple "what-if" questions or advanced simulation-based modelling, such as Monte Carlo analysis.

5.6 Evaluation challenges

This chapter presents the main challenges for the national impact and economic evaluation of No One Left Behind, as identified through stakeholder interviews, data mapping, and development of the evaluation plan. The challenges are organised into two broad categories: data-related and methodology-related.

For each challenge, we offer a brief description, an estimated level of impact if left unaddressed, and potential mitigating actions.

5.6.1 Data challenges

These challenges include data gaps, robustness concerns, and caveats related to the datasets we expect to use in the evaluation.

• DRT: As detailed in the Interview Findings section, the DRT has several limitations and robustness concerns. In summary: (i) the interpretation of definitions and questions can differ across LEPs; (ii) reliance on self-reporting can introduce inaccuracies; (iii) repeat engagement with No One Left Behind is not recorded as new entries; (iv) not all data fields were available from the first iteration of DRT; and (v) there can be month-long delays between service provision and data logging.
  • Level of impact: The impact of low data quality and the reliability of the DRT on the No One Left Behind evaluation is expected to be high. This is because the DRT is the only way to identify No One Left Behind participants and a key source of information on participants' outcomes.
  • Mitigation options: Our proposed plan makes use of administrative datasets to reduce reliance on DRT. Specifically, we propose using the DRT only to identify No One Left Behind participants and then using the NI number to track their outcomes in other datasets. However, there would still be risks associated with repeated participant engagement, which we expect to be mitigated by the upcoming “restarts” project (see the Interviews section).
  • Mitigation options: We recommend combining several datasets to ensure maximum coverage. Using the NI number of No One Left Behind participants from the DRT, the evaluator can get (i) earnings and employment data from the HMRC RTI; (ii) education data from the APS; (iii) benefit-claimant data from the RAPID; and (iv) health data from the APS. For any missing data, we rely on contribution analysis, which can draw on quantitative analysis and indicative counterfactuals based on survey information, but these outcomes then cannot be used for robust QED.
  • Mitigation options: Our evaluation recommendations (outlined in the next chapter) include using the unreached non-participant control group, under a “no-employability support” counterfactual. As a result, any impacts achieved through employability support should be considered, regardless of their exact funding source. However, given the overlaps between funding and difficulties in separating costs of different activities within blended funded models, there may be a case for taking a subset of cost data and doing a deep dive to understand what is included, etc. (where they may share infrastructure, etc. - maybe separate piece of work)
  • Mitigation options: If most of the missing participants are from a single area or make use of a specific service, more consideration should be given to the propensity score weighting (see the Evaluation Considerations section). In particular, the evaluator should ensure that the weights applied to the control and treatment groups account for any systematic differences observed among the missing population, thus maximising the likelihood that we can reliably proxy their outcomes.
• Administrative datasets: The refined shortlist in the Data Mapping and Assessment section includes four datasets, in addition to the DRT. However, no single dataset captures all necessary information.
  • Level of impact: The impact of the missing data items in the four datasets on the No One Left Behind evaluation is expected to be moderate, as most shortlisted datasets are missing one or two of the required data.
  • Mitigation options: We recommend combining several datasets to ensure maximum coverage. Using the NI number of No One Left Behind participants from the DRT, the evaluator can get (i) earnings and employment data from the HMRC RTI; (ii) education data from the APS; (iii) benefit-claimant data from the RAPID; and (iv) health data from the APS. For any missing data, we rely on contribution analysis, which can draw on quantitative analysis and indicative counterfactuals based on survey information, but these outcomes then cannot be used for robust QED.
  • Mitigation options: Our evaluation recommendations (outlined in the next chapter) include using the unreached non-participant control group, under a “no-employability support” counterfactual. As a result, any impacts achieved through employability support should be considered, regardless of their exact funding source. However, given the overlaps between funding and difficulties in separating costs of different activities within blended funded models, there may be a case for taking a subset of cost data and doing a deep dive to understand what is included, etc. (where they may share infrastructure, etc. - maybe separate piece of work)
  • Mitigation options: If most of the missing participants are from a single area or make use of a specific service, more consideration should be given to the propensity score weighting (see the Evaluation Considerations section). In particular, the evaluator should ensure that the weights applied to the control and treatment groups account for any systematic differences observed among the missing population, thus maximising the likelihood that we can reliably proxy their outcomes.
• Separate data on No One Left Behind funding: While LEP reporting distinguishes between funding from No One Left Behind and other sources, in practice, services are often co-funded and delivered through blended funding models. Additionally, the extent of this blending varies considerably across different LEPs.
  • Level of impact: The impact of this lack of clear separation between funding sources on evaluation is expected to be high. Without any mitigating actions, any identified impact should not be attributed solely to No One Left Behind funding, but to the total funding provided to LEPs.
  • Mitigation options: Our evaluation recommendations (outlined in the next chapter) include using the unreached non-participant control group, under a “no-employability support” counterfactual. As a result, any impacts achieved through employability support should be considered, regardless of their exact funding source. However, given the overlaps between funding and difficulties in separating costs of different activities within blended funded models, there may be a case for taking a subset of cost data and doing a deep dive to understand what is included, etc. (where they may share infrastructure, etc. - maybe separate piece of work)
  • Mitigation options: If most of the missing participants are from a single area or make use of a specific service, more consideration should be given to the propensity score weighting (see the Evaluation Considerations section). In particular, the evaluator should ensure that the weights applied to the control and treatment groups account for any systematic differences observed among the missing population, thus maximising the likelihood that we can reliably proxy their outcomes.
• HMRCRTI: Based on our stakeholder interviews, we expect approximately 90% of No One Left Behind participants to be potentially identifiable in the HMRC RTI dataset. This means there will be no earnings and employment data for the remaining 10%, at least.
• Level of impact: The expected impact is moderate, as the missing population represents a minority. However, given that the basis for linkage is information on multiple disadvantages, the missing 10% of participants may differ substantially or systematically from the remaining No One Left Behind participants. As a result, the missing population may differ enough to have significantly different outcomes than the rest.
• Mitigation options: If most of the missing participants are from a single area or make use of a specific service, more consideration should be given to the propensity score weighting (see the Evaluation Considerations section). In particular, the evaluator should ensure that the weights applied to the control and treatment groups account for any systematic differences observed among the missing population, thus maximising the likelihood that we can reliably proxy their outcomes.

5.6.2 Methodological challenges

These challenges relate to the core methodological aspects of conducting the evaluation, including the measurement and monetisation of outcomes, and the identification of the control group.

• Harder to capture outcomes: The ToC section presents the identified impacts of No One Left Behind, including reduced employment rates, reduced reliance on unemployment benefits, and improved health and wellbeing. However, as detailed in the “estimation of outcomes” chapter, the data mapping exercise has uncovered gaps in some of these outcomes that are also difficult to quantify and monetise.
  • Level of impact: The impact of data gaps in some of the outcomes of interest ranges from moderate to high. Health, wellbeing, and education data gaps do not affect the feasibility of the overall evaluation, while benefit claims are a key factor in identifying a robust control group.
  • Mitigation options:
    • Health data are available in the NHS Scotland datasets. However, these datasets use CHI instead of NI numbers. As a result, even if a data access agreement were established, a separate exercise to link CHI and NI numbers would still be required. APS/LFS surveys capture some data on disability and long-term conditions, while RAPID can be used as a proxy for health conditions for which benefits are claimed. However, neither option constitutes an exhaustive list of health conditions.
    • Wellbeing data is captured in the Scottish Health Survey. However, the survey does not use NI numbers, so a data agreement and a dedicated linkage project would need to be commissioned to enable its use for tracking wellbeing data. An alternative for No One Left Behind participants could be new primary data collection (e.g. surveys) or expansion of the user satisfaction survey currently underway. However, stakeholders interviewed have highlighted that LEPs are already struggling to meet their current data reporting requirements and have advised against new data collection.
    • Education: Education outcomes are recorded in the DRT, but robustness concerns were raised by stakeholders. Furthermore, it is best practice to use a single dataset for both the control and treatment groups. As a result, the APS/LFS surveys will need to be used for both groups. However, these surveys have significant limitations due to the lack of NI numbers (see the Economic Evaluation section for more details). All things considered, education is likely challenging to evaluate through QED approaches and would be better suited to a theory-based approach supported by quantitative evidence.
    • Benefit claims: The main source of information for benefit claims is the RAPID dataset. While RAPID uses NI numbers, new data access agreements are required to establish linkage. Social Security Scotland also offers data on claims of devolved benefits. While it does not include UK-administered benefits, data access agreements could be more straightforward, allowing this dataset to serve as a proxy for broader benefit claims.
    • Mitigation options: While the No One Left Behind design and the variability in treatment groups minimise the selection bias risk, it can never be completely eliminated. Our proposed evaluation design further reduces it by: (i) addressing constant differences between treatment and control by looking at changes and not levels; (ii) carefully weighting treatment and control individuals to make them as comparable as possible and hence making the parallel trend assumption even more plausible. If any differences persist due to unobserved factors, theory-based approaches should be used.
    • Mitigation options: As previously mentioned, our proposed evaluation design of doubly robust DiD mitigates sampling bias risks. This is achieved through a set of “local area control” variables, such as local unemployment rates, wages, sectoral distribution, urbanisation levels, and demographic characteristics. Together, these variables constitute a statistical description of specific areas or groups of areas that can then be used as weights in the inverse weighting process (covered in the Impact Evaluation section), achieving a national evaluation with local characteristics.
    • Mitigation options: This limitation can be addressed by tailoring the available data to the Scottish context. For example, avoided hospitalisation costs can be adjusted based on wage or overall price differentials between Scotland and the rest of the UK. Furthermore, the estimates can be sense checked and adjusted by engaging with the relevant stakeholders.
    • Mitigation options: While we advise against this separation, as it would also limit the control and treatment group sizes for each strand, there are some options if the evaluator wanted to pursue this option. Most notably, demographic characteristics of No One Left Behind participants could be used as a proxy to validate the reported spending breakdowns, using data on family status (or number of children) for the parental strand, and long-term health conditions or disabilities for the specialist support strand. The remaining population would be assumed to be serviced by the all-age support strand.
    • Mitigation options: While the No One Left Behind design and the variability in treatment groups minimise the selection bias risk, it can never be completely eliminated. Our proposed evaluation design further reduces it by: (i) addressing constant differences between treatment and control by looking at changes and not levels; (ii) carefully weighting treatment and control individuals to make them as comparable as possible and hence making the parallel trend assumption even more plausible. If any differences persist due to unobserved factors, theory-based approaches should be used.
    • Mitigation options: As previously mentioned, our proposed evaluation design of doubly robust DiD mitigates sampling bias risks. This is achieved through a set of “local area control” variables, such as local unemployment rates, wages, sectoral distribution, urbanisation levels, and demographic characteristics. Together, these variables constitute a statistical description of specific areas or groups of areas that can then be used as weights in the inverse weighting process (covered in the Impact Evaluation section), achieving a national evaluation with local characteristics.
    • Mitigation options: This limitation can be addressed by tailoring the available data to the Scottish context. For example, avoided hospitalisation costs can be adjusted based on wage or overall price differentials between Scotland and the rest of the UK. Furthermore, the estimates can be sense checked and adjusted by engaging with the relevant stakeholders.
    • Mitigation options: While we advise against this separation, as it would also limit the control and treatment group sizes for each strand, there are some options if the evaluator wanted to pursue this option. Most notably, demographic characteristics of No One Left Behind participants could be used as a proxy to validate the reported spending breakdowns, using data on family status (or number of children) for the parental strand, and long-term health conditions or disabilities for the specialist support strand. The remaining population would be assumed to be serviced by the all-age support strand.
• Selection bias for control group: The most appropriate control group, as suggested earlier, consists of individuals who share similar observable characteristics with No One Left Behind participants and could benefit from the programme, but who have not taken part. However, these individuals may differ from one another in unobserved characteristics, such as personal circumstances and motivation. These differences could affect their likelihood of participating in No One Left Behind and their outcomes, making it difficult to attribute any observed outcome gaps specifically to the approach.
  • Level of impact: The impact of selection bias in choosing the individuals for the control group on the evaluation is expected to be low. Based on stakeholder interviews, each LEP targets individuals with varying degrees of work readiness. This means that some LEPs identify and offer support mostly to those already closer to the labour market, while others strive to support those with the greatest needs. This variability in reach, along with the variety of services offered, ensures that, at the national level, No One Left Behind participants represent a wide range of circumstances, motivations, and other unobserved characteristics that cannot be explicitly accounted for in the control group.
  • Mitigation options: While the No One Left Behind design and the variability in treatment groups minimise the selection bias risk, it can never be completely eliminated. Our proposed evaluation design further reduces it by: (i) addressing constant differences between treatment and control by looking at changes and not levels; (ii) carefully weighting treatment and control individuals to make them as comparable as possible and hence making the parallel trend assumption even more plausible. If any differences persist due to unobserved factors, theory-based approaches should be used.
  • Mitigation options: As previously mentioned, our proposed evaluation design of doubly robust DiD mitigates sampling bias risks. This is achieved through a set of “local area control” variables, such as local unemployment rates, wages, sectoral distribution, urbanisation levels, and demographic characteristics. Together, these variables constitute a statistical description of specific areas or groups of areas that can then be used as weights in the inverse weighting process (covered in the Impact Evaluation section), achieving a national evaluation with local characteristics.
  • Mitigation options: This limitation can be addressed by tailoring the available data to the Scottish context. For example, avoided hospitalisation costs can be adjusted based on wage or overall price differentials between Scotland and the rest of the UK. Furthermore, the estimates can be sense checked and adjusted by engaging with the relevant stakeholders.
  • Mitigation options: While we advise against this separation, as it would also limit the control and treatment group sizes for each strand, there are some options if the evaluator wanted to pursue this option. Most notably, demographic characteristics of No One Left Behind participants could be used as a proxy to validate the reported spending breakdowns, using data on family status (or number of children) for the parental strand, and long-term health conditions or disabilities for the specialist support strand. The remaining population would be assumed to be serviced by the all-age support strand.
• Local variation: Stakeholders have consistently noted throughout the project the significant variations in the funding, design, and delivery of No One Left Behind activities across local areas. This was mentioned as one of the approach’s biggest strengths and challenges at the same time.
  • Level of impact: The local variations have a low impact if the evaluation is done on a national level. However, there is a risk of selection bias if our control group undersamples specific areas that may differ significantly from the rest of the population in characteristics and outcomes.
  • Mitigation options: As previously mentioned, our proposed evaluation design of doubly robust DiD mitigates sampling bias risks. This is achieved through a set of “local area control” variables, such as local unemployment rates, wages, sectoral distribution, urbanisation levels, and demographic characteristics. Together, these variables constitute a statistical description of specific areas or groups of areas that can then be used as weights in the inverse weighting process (covered in the Impact Evaluation section), achieving a national evaluation with local characteristics.
  • Mitigation options: This limitation can be addressed by tailoring the available data to the Scottish context. For example, avoided hospitalisation costs can be adjusted based on wage or overall price differentials between Scotland and the rest of the UK. Furthermore, the estimates can be sense checked and adjusted by engaging with the relevant stakeholders.
  • Mitigation options: While we advise against this separation, as it would also limit the control and treatment group sizes for each strand, there are some options if the evaluator wanted to pursue this option. Most notably, demographic characteristics of No One Left Behind participants could be used as a proxy to validate the reported spending breakdowns, using data on family status (or number of children) for the parental strand, and long-term health conditions or disabilities for the specialist support strand. The remaining population would be assumed to be serviced by the all-age support strand.
• Monetisation and valuation challenges: The monetisation of outcomes identified in the impact evaluation relies primarily on UK or England estimates. If these estimates are not representative of Scotland, they may introduce bias into benefit valuation and affect the reliability of economic evaluation.
  • Level of impact: The impact of the lack of Scotland-specific estimates for monetisation on the evaluation is expected to be low. Using England or UK-wide estimates is a generally accepted method of addressing Scotland-specific data gaps.
  • Mitigation options: This limitation can be addressed by tailoring the available data to the Scottish context. For example, avoided hospitalisation costs can be adjusted based on wage or overall price differentials between Scotland and the rest of the UK. Furthermore, the estimates can be sense checked and adjusted by engaging with the relevant stakeholders.
  • Mitigation options: While we advise against this separation, as it would also limit the control and treatment group sizes for each strand, there are some options if the evaluator wanted to pursue this option. Most notably, demographic characteristics of No One Left Behind participants could be used as a proxy to validate the reported spending breakdowns, using data on family status (or number of children) for the parental strand, and long-term health conditions or disabilities for the specialist support strand. The remaining population would be assumed to be serviced by the all-age support strand.
• Separation of analysis by funding strand: As previously mentioned, the No One Left Behind provision is funded and structured around three key strands, namely all age, parental, and specialist employability support. While some AIPs and financial reports break down information by strand, discussions with stakeholders have raised concerns about the robustness of these breakdowns, as services frequently support mixed participant groups, making it difficult to directly and consistently attribute spending to specific funding strands.
  • Level of impact: The anticipated impact for the overall evaluation is low, as the funding breakdowns do not affect the final results, only the ability to separate them by strand. From a national evaluation perspective, the precise split between strands is not significant, as all funding comes from the same source.
  • Mitigation options: While we advise against this separation, as it would also limit the control and treatment group sizes for each strand, there are some options if the evaluator wanted to pursue this option. Most notably, demographic characteristics of No One Left Behind participants could be used as a proxy to validate the reported spending breakdowns, using data on family status (or number of children) for the parental strand, and long-term health conditions or disabilities for the specialist support strand. The remaining population would be assumed to be serviced by the all-age support strand.