No One Left Behind: evaluability assessment (national impact and economic evaluation)

4. Methodology

This chapter outlines the methodology used in the impact and economic evaluability assessment of No One Left Behind. In summary, our approach consisted of the following phases:

• Scoping and desk-based review, during which we reviewed the existing evidence on No One Left Behind to build our understanding of the approach and develop a final project plan.
• Development of a data mapping and assessment framework to systematically log all relevant data sources and evaluate their usefulness for the potential evaluations.
• Interviews with No One Left Behind stakeholdersand regular meetings with a Research Advisory Group (RAG) ^[2] to validate and expand our understanding of policy priorities, the No One Left Behind approach, and the data landscape.
• Refinement of the existing logic model into a Theory of Change (ToC) explaining how No One Left Behind operates.
• Development of an evaluation plan based on their theoretical and practical viability.

Below, we offer additional details about our approach to stages 2-5.

4.1 Data assessment framework and mapping

Our desk-based review and sources provided by the SG formed the basis for an initial long list of potential data sources. The framework was then used to refine the long list into a shortlist of the most robust and useful data for a Scotland-wide impact and economic evaluation of No One Left Behind.

The data assessment framework provided a standardised basis for tracking and comparing datasets to determine their suitability for the evaluation. Specifically, it was first used to exclude irrelevant datasets and then to rank the most promising ones.

4.1.1 Exclusion criteria

Our exclusion criteria were based on both established best practices for what constitutes a robust dataset and No One Left Behind-specific considerations to ensure relevance to national impact and economic evaluation. In particular, we employed the following criteria:

Geographical coverage: Datasets that did not cover individuals in Scotland were not included, as evaluation of No One Left Behind requires data that is representative of the Scottish population.

Level of disaggregation: We considered only individual-level datasets to systematically track outcomes for each No One Left Behind participant. Aggregate-level impact evaluation does not allow for robust identification of impacts, as there may be competing effects (e.g. increased benefit uptake due to support with benefit applications, combined with reduced benefit claims due to lower unemployment). Furthermore, without individual-level data, it is not possible to establish a credible counterfactual (see the evaluation chapter for more details).

Time coverage: To assess the impact of No One Left Behind, the evaluator needs robust data both before and after the approach's introduction to establish credible pre-trends for both No One Left Behind participants and the general population. As a result, any datasets not covering at least the period of 2019-2026 were not considered.

Variables of interest: We established a list of key No One Left Behind outcomes based on our desk-based research, approach documentation, discussions with key stakeholders, and our refined ToC. Examples included earnings, employment, progression, qualifications, and health and wellbeing. Any data sources that did not provide at least partial coverage of some of these outcomes were excluded from further consideration.

4.1.2 Shortlisting and ranking data sources

After excluding all irrelevant datasets based on the aforementioned criteria, we created a long list of relevant data sources (presented in Annex 2). We then conducted a more thorough examination of these datasets to identify the most promising, which were included on a shortlist. Finally, the shortlisted datasets were ranked based on their alignment with the criteria outlined below.

Overall quality and robustness: Datasets with established data quality checks, reliable documentation, low levels of missing information, and significant sample sizes were prioritised.

Alignment to Shared Measurement Framework (SMF): In recognition of the Scottish Government’s strategic priorities and the established outcomes of interest for No One Left Behind, we prioritised datasets that aligned with at least one of the SMF thematic areas, namely:

• Reach, including data on equality characteristics (e.g., age, disability, health conditions) and personal circumstances (e.g., homelessness, criminal convictions, parental status).
• Progression, covering information on employment and education outcomes over time.
• Skills assignment, including data on accreditations and certifications.
• Value of services, consisting of outcomes such as income, health, and wellbeing.

4.2 Stakeholder interviews

In parallel with the data assessment, our team conducted 12 interviews with 15 stakeholders in the employability sector. These interviews were designed to (i) identify and agree on the key questions that any future Scotland-level impact or economic evaluation should address; and (ii) establish a clear understanding of the data available to support the evaluation. The interviews also provided insight into how No One Left Behind is implemented in practice and helped validate our understanding of both the approach and its wider policy context, ensuring that our feasibility assessment accurately reflected No One Left Behind's operational reality.

The interviews were semi-structured and conducted online via Microsoft Teams. Stakeholder selection was facilitated by the Scottish Government, which shared relevant contact details and, in some cases, did direct outreach. Interviews lasted approximately 45 minutes and included individual or small-group discussions (up to 3 participants). The fieldwork period ran from 16 to 24 March 2026, scheduled to accommodate the pre-election period.

A single discussion guide was used (Annex 3), tailored to stakeholder type across (i) central policy and Local Employability Partnership representatives; and (ii) data experts. Both strands covered common areas, including priority questions for future impact and economic evaluation of No One Left Behind, as well as questions about how the approach functions. However, their focus differed based on each group’s expertise, detailed below.

4.2.1 Central policy and Local Employability Partnership representatives

Seven interviews were conducted with eight stakeholders from: (i) the Scottish Government; (ii) TSI Scotland Network; and (iii) LEP employability leads. The topics addressed included:

• Impact evaluation priorities, such as the factors influencing No One Left Behind’s success, service user outcomes, and counterfactual scenarios (i.e., what might have happened in the absence of No One Left Behind).
• Economic evaluation priorities, including the most important outcomes for assessing value for money (VfM), the preferred evaluation time horizon, and the availability of expenditure data.
• Understanding of No One Left Behind operational reality, including how the referral process works, patterns in No One Left Behind interventions, and differences in the interpretation of data definitions under the data reporting template.

4.2.2 Data experts

Five interviews were conducted with seven stakeholders, comprising relationship managers, cost specialists, a Scottish Government employability statistics lead, and LEP data experts. The topics addressed included:

• Data availability, including (i) an assessment of the quality and usefulness of any public datasets that could inform the evaluation; (ii) understanding the format and limitations of data provided to the Scottish Government as part of statistical and financial reporting; and (iii) identifying locally held data that could enhance the analysis.
• Economic evaluation priorities, focusing on the availability of No One Left Behind expenditure data held by the Scottish Government and data on other Scottish Government-funded employability activities held by LEPs.

4.3 Refinement of the existing logic model

Through the desk-based review and the stakeholder interviews, we developed an understanding of the inputs, expected outputs, and outcomes of the No One Left Behind approach. These were then used to develop a Refined Theory of Change, which maps how No One Left Behind operates and creates impacts.

A ToC serves as a key tool for both impact and economic evaluations, supporting the identification of outcomes to quantify and monetise. It also provides a basis for exploring potential causal links between No One Left Behind-funded activities and outcomes across multiple dimensions, including employment, income maximisation, household outcomes, equitable employment, and health and wellbeing.

The revised ToC builds on an original logic model developed at the inception of No One Left Behind and subsequently updated in 2025 to align with the approach’s strategic plan. This earlier model set out an approach logic across two levels of inputs and outputs (Scottish Government and LEP) and three levels of outcomes (strategic, service delivery, and service user). While this provided a useful high-level overview of the approach, it was largely process-focused and offered limited detail on funded activities and the pathways through which change was expected to occur. Notably, outcomes were articulated primarily at the strategic and service delivery levels, with less emphasis on the mechanisms driving change for service users. The separation of inputs/outputs and outcomes across multiple levels also created challenges for developing a coherent framework suitable for national-level impact and economic evaluation.

As part of this revision, the logic model’s structure was streamlined by consolidating Scottish Government inputs (e.g. funding, strategy, governance, and implementation tools) and merging LEP inputs and activities into a clearer delivery chain. The revised ToC was first informed by a detailed review of a sample of LEPs’ Annual Investment Plans (AIPs), aimed at covering a representative breadth of activities and geographies. This review was key in supporting the development of a typology of funded activities and identifying the range of outcome types being delivered in practice. We then triangulated our findings through stakeholder interviews, which helped ensure a more accurate reflection of No One Left Behind’s operational reality and the expected outcomes.

The development of the ToC was an iterative process that began with a presentation to key stakeholders from the RAG, followed by a round of feedback. This helped validate the identified causal pathways and ensure that the model reflects the key areas of interest.

4.4 Development of an evaluation plan

Using the refined ToC and our findings from the desk-based review and stakeholder engagement, we developed a plan for the national economic and impact evaluation of No One Left Behind.

Impact evaluation refers to the process of estimating the causal effect of a policy, programme, or intervention. It seeks to determine what difference the intervention made by comparing observed outcomes with an estimate of what would have happened in its absence (the counterfactual). Economic evaluation, in turn, seeks to quantify and monetise the identified impacts to assess whether the examined interventions represent VfM for the Exchequer and wider society. It seeks to compare the costs and benefits of an intervention using cost-effectiveness or Cost-Benefit analysis (explained in detail in the Economic Evaluation section).

Our approach consisted of two layers: (i) exploring the theoretical validity of different evaluation designs, and (ii) assessing the practicality and feasibility of each design. For the first layer, we explored a wide range of quasi-experimental designs (QED), theory-based methods, and mixed-methods approaches. These were then assessed based on robustness, data availability, and No One Left Behind-specific challenges and opportunities. Once we identified all options that were theoretically viable, we assessed their practical feasibility based on factors such as timeliness, resources, data-collection burden, and potential counterfactuals.

The counterfactual provides the future evaluator with a credible ‘do-nothing’ scenario to compare the outcomes achieved by No One Left Behind and establish attribution of success to the funded services. To develop different options for the impact evaluation, our team adopted best practices in policy evaluation as set out in the HM Treasury Green Book and Magenta Book. A detailed list of all evaluation approaches and counterfactuals considered is presented in the Findings section.

4.5 Limitations

While our approach aligns with established best practices and the Green Book and Magenta Book evaluation guidance, it is worth noting that our findings are subject to several limitations, outlined below:

• Stakeholder engagement was limited, both in terms of coverage within the Scottish Government and at the LEP level. Due to time limitations placed by the pre-election period, we were able to engage with only a subset of partnerships (4 out of 32). While this level of engagement is considered sufficient for the purposes of an evaluability assessment supporting, further engagement with LEPs in a full evaluation may reveal greater variability in service design, funding, and delivery.
• The document review of No One Left Behind reporting outputs was restricted to a single example of the grant offer and finance report template, alongside four populated Annual Investment Plans selected to reflect variation in size, rurality, and deprivation. A review of additional templates and plans could reveal further variation in reporting practices and funding detail.
• The data reporting template was reviewed only in its template form, without access to any populated data. While this allows assessment of structure and intended use, analysis of completed templates would be needed to fully understand data quality, consistency, and completeness in practice.
• Finally, it is worth noting that the scope of this project was to assess the feasibility of a national impact and economic evaluation of No One Left Behind. As a result, we did not focus on or explicitly examine a process evaluation.