No One Left Behind: evaluability assessment (national impact and economic evaluation)

1. Executive Summary

1.1 Background

No One Left Behind was launched by the Scottish Government in 2019 as a locally led employability support model aimed at replacing the previous centralised system. It focuses on providing flexible, person-centred services to enhance labour market participation for individuals facing employment barriers, while also addressing broader issues like child poverty and health inequalities. No One Left Behind has grown to include various funding streams and delivery models, such as targeted support for parents and specialised assistance for people with disabilities and long-term health conditions. Local Employability Partnerships (LEPs) adapt their services to meet local needs, though this flexibility poses challenges for national-level evaluations.

Given the importance of No One Left Behind to Scotland's employability landscape, Alma Economics was commissioned by the Scottish Government to conduct an evaluability assessment for a future national impact and economic evaluation of No One Left Behind. This project, therefore, assesses the feasibility of robust evaluation, identifies suitable methodological approaches, and outlines data and analytical requirements for future evaluations.

1.2 Methodology

The evaluability assessment combined several complementary methodological strands. First, a desk-based review was undertaken to examine the policy context and existing evidence base. Second, a structured data-mapping and assessment exercise was conducted to identify datasets capable of supporting impact and economic evaluations. This included an assessment of data coverage, quality, linkage potential, and alignment with No One Left Behind outcomes and the Shared Measurement Framework. Third, stakeholder engagement was undertaken through interviews with Scottish Government representatives, LEP leads, data specialists, and wider sector stakeholders to understand the operational realities of No One Left Behind delivery, the availability and robustness of evidence, and key priorities for future evaluation. The existing No One Left Behind logic model was then refined into a revised Theory of Change (ToC) to provide a clearer articulation of the pathways through which No One Left Behind activities are expected to generate outcomes and impacts. Finally, we identified all theoretically feasible evaluation designs. For impact evaluation, this included experimental, quasi-experimental, and theory-based methods. For economic evaluation, this focused on the feasibility of a cost-benefit analysis as the most robust option. These were then assessed for their suitability based on HM Treasury Green Book and Magenta Book guidance and informed by the learning and information collected in the previous steps. Recommended evaluation methods, approaches and challenges were then set out.

1.3 Findings

1.3.1 Stakeholder interviews

The assessment identified several important considerations for the future evaluation of No One Left Behind. Most significantly, the locally tailored nature of delivery creates substantial variation in how services are commissioned, targeted, and implemented across Scotland. While this reflects the approach’s intended design, it limits the extent to which outcomes can be interpreted as arising from a single, uniform intervention. Stakeholders consistently highlighted that any future evaluation will therefore need to recognise No One Left Behind as a system-level approach rather than a standardised programme model. This has implications both for the interpretation of impact estimates and for the selection of appropriate counterfactuals and comparison groups.

1.3.2 Impact evaluation

The most viable options for a future impact evaluation are quasi-experimental (QED) and theory-based approaches. In particular, propensity score matching (PSM) and difference-in-differences (DiD) approaches were identified as theoretically feasible, subject to the availability of sufficiently robust and linkable administrative data. By combining elements from both, we proposed a “doubly robust” DiD approach for employment and earnings impacts.

Contribution analysis and realist evaluation were also found to be well suited to the No One Left Behind context, particularly given the approach’s complexity, local variation, and emphasis on system change and person-centred delivery. These theory-based approaches would be better suited to education, health, and wellbeing outcomes, given the identified data gaps and challenges. A quasi-experimental approach could be possible for these outcomes as well, through the use of proxies and assumptions (at the cost of robustness) or new primary data collection (entailing financial costs and a data-collection burden).

The assessment, therefore, concludes that a mixed-methods approach combining quantitative impact estimation with theory-based and qualitative analysis is likely to provide the most credible and policy-relevant evaluation framework.

1.3.3 Counterfactual and control group

The assessment also identified important considerations regarding counterfactuals and control groups. Several possible comparator groups were explored, including unreached eligible non-participants, participants in other employability programmes, disengaged participants, and recipients of previous national employability provision such as Fair Start Scotland (FSS). However, each option presents methodological and practical limitations relating to selection bias, overlapping provision, and differences in programme design and delivery. Out of the examined options, the most robust counterfactual is “unreached non-participants”, enhanced by the variable weighting of individuals under the “doubly robust” DiD approach.

1.3.4 Economic evaluation

An economic evaluation of No One Left Behind is also possible. Based on our evaluability assessment, we suggest conducting a Social Cost-Benefit Analysis that includes both the Exchequer (fiscal) and societal perspectives. In summary, the fiscal benefits include: (i) reduction in social security payments; (ii) increase in the number of people paying taxes; (iii) increase in tax contributions from the existing tax base; (iv) increase in taxes due to an increase in lifetime earnings from improved education; (v) reduced NHS costs due to better health and wellbeing.

In contrast, the social benefits include: (i) an increase in economic output due to more people entering employment; (ii) an increase in economic output due to higher earnings; (iii) an increase in economic output due to higher lifetime earnings from better education; (iv) health or wellbeing benefits.

The calculation of benefits will primarily rely on established estimates of the monetary value of each impact, while cost information can be sourced from Scottish Government and LEPs records. To increase the robustness of the findings, the evaluator should adjust for several factors, such as uncertainty, as outlined in the Green Book guidance (e.g., deadweight, optimism bias, social discounting, stress-testing).

1.3.5 Data availability

Data availability presents both an opportunity and a challenge for future evaluations. The data reporting template (DRT) is the primary source for identifying No One Left Behind participants and tracking service outcomes. However, stakeholders raised concerns about inconsistent reporting, evolving definitions, data lag, and limited coverage of softer outcomes. Despite these issues, discussions on administrative data linkage, particularly with HMRC Real Time Information (RTI), are seen as a potentially significant development for evaluation. This linkage would enhance employment and earnings analyses and aid in tracking participant outcomes over the long term. However, proposed data linkage agreements cover only programme participants, limiting their usefulness for establishing a control group. Full data access agreements with HMRC for the RTI and the Department for Work and Pensions (DWP) for the Registration and Population Interaction database (RAPID) would present a significant opportunity that could greatly enhance the robustness of a control group, and hence the impact evaluation. Health outcomes may also be quantifiable, conditional on access to NHS Scotland datasets and linkage of Community Health Index (CHI) and National Insurance (NI) numbers.

In summary, full access (i.e. coverage of both treatment and control groups) in HMRC RTI, DWP RAPID, and NHS Scotland datasets would significantly enhance the robustness of the evaluation. If full access to HMRC RTI is not achieved, QED impact and economic evaluations would likely not be possible, leaving only theory-based approaches as a feasible option.

1.3.6 Challenges and limitations

It is worth noting that the impact and economic evaluation of No One Left Behind come with significant challenges and limitations. The ones with the highest anticipated impact are:

• The data reporting template was only reviewed in template form and not with populated data. A greater understanding of data quality, consistency and completeness in practice is needed. We propose only using it to identify participants, rather than to estimate any outcomes.
• Some outcomes, including system-wide operation changes, health, wellbeing, and education, are difficult to quantify and monetise. If data linkage could be achieved with NHS Scotland datasets, health and wellbeing outcomes could potentially be quantified. Education would likely be approached qualitatively, supported by some high-level quantitative estimates. Wider operational outcomes should be examined in a separate process evaluation.
• Services are commonly delivered through blended funding arrangements, making it difficult to isolate outcomes attributable solely to No One Left Behind funding. To address this, we propose a counterfactual of ‘no employability support’, in which all forms of support are included, regardless of funding source. This approach avoids the need to separate outcomes by funding source.
• The main strength of the No One Left Behind Approach is its flexibility and tailoring to the local context. However, this makes it challenging to evaluate nationally. Our doubly robust difference-in-differences approach can partially account for differences in local areas.

1.3.7 Conclusion

Overall, the evaluability assessment concludes that a Scotland-level impact and economic evaluation of No One Left Behind is feasible, provided that future evaluation activity is designed to reflect the approach’s complexity, delivery variation, and evolving data infrastructure. The strongest evaluation design would combine linked administrative datasets (i.e., HMRC RTI and DWP RAPID), quasi-experimental impact estimation (i.e., doubly robust DiD), and theory-based approaches (i.e., contribution analysis with realist elements), supported by qualitative evidence. Future evaluation work should also consider complementary process evaluation activities to better understand implementation differences across areas and the mechanisms through which outcomes are achieved.

The following flow diagram summarises our proposed workflows for the overall evaluation of No One Left Behind. Each step is explained in more detail in Chapter 6.