No One Left Behind: evaluability assessment (national impact and economic evaluation)

6. Evaluation plan

This chapter brings together the findings of the evaluability assessment to offer an actionable plan for the impact and economic evaluation of No One Left Behind. A visualisation of the plan is included as a flowchart at the end of this chapter.

It is worth noting that, while this is outside the scope of this assessment, the majority of the stakeholders we engaged with expressed a strong preference for a process evaluation to be conducted alongside the impact and economic evaluations. In such a case, the process evaluation should be conducted before the impact evaluation (both theory-based and QED approaches), which in turn should be carried out before the economic evaluation, as each stage builds on the evidence generated in the previous one.

The process evaluation helps to establish whether the programme has been implemented as intended, providing essential context for interpreting impact estimates and identifying any variation in delivery across areas. It will also examine the system-wide and operational outcomes identified in the ToC. The impact evaluation can then assess the extent to which observed outcomes can be attributed to the programme. Finally, the economic evaluation relies on robust estimates of impact to monetise benefits and assess VfM, making it appropriate to conduct it after the impact analysis is complete.

6.1 Impact evaluation

Our proposal for the impact evaluation is a mixed-methods approach that involves a “doubly robust” DiD, complemented by contribution analysis.

6.1.1 Step 1: Finalise data access agreements and linkage projects

While initial agreement on data linkage with HMRC RTI has been established for the treatment group, additional agreements and linkage projects will be required to capture all identified No One Left Behind impacts. These include: (i) data on comparable individuals for the control group from HMRC RTI; (ii) benefit-claims data from RAPID and Social Security Scotland; (iii) healthcare data from NHS Scotland datasets; and (iv) wellbeing data from the Scottish Health Survey.

This is a crucial and likely time- and resource-intensive step in the evaluation plan. Without full access to at least HMRC RTI, QED impact and economic evaluation approaches would likely not be possible.

6.1.2 Step 1.2: Contribution analysis with realist elements

Once data access agreements and linkage projects are finalised, a theory-based evaluation can be conducted. This can be used either in conjunction with a quasi-experimental evaluation to provide context for the estimates or as a mitigation to capture any impacts that cannot be quantified.

Based on our evaluability assessment, we expect the theory-based evaluation to be used primarily for health and wellbeing impacts. While the datasets mentioned above include relevant information, gaining access and establishing data linkage (especially with NHS Scotland, which uses CHI instead of NI numbers) is likely to be prohibitively time- and resource-intensive.

The primary theory-based method will likely be contribution analysis. This would leverage the refined ToC developed as part of this project and seek to establish a credible "contribution claim" by demonstrating that an intervention's activities were implemented as intended and that the evidence chain of results aligns with the refined ToC. To that end, it will be key to conduct additional stakeholder engagement (e.g. surveys, interviews, focus groups) with additional LEPs and LAs to ensure the impacts identified from the sample AIPs are representative of all areas.

As part of the contribution analysis, the evaluator could take a “realist approach” by identifying the conditions under which the impacts materialise and examining whether these conditions hold. This approach would function as a high-level realist evaluation, without investing substantial resources.

6.1.3 Step 2: Establishing the treatment and control groups

Once all data access agreements are in place, the evaluator should establish the counterfactual and control group. While several control groups were examined, we suggest using “unreached non-participants” as the control.

This group includes individuals who share similar observable characteristics with No One Left Behind participants and could potentially benefit from No One Left Behind support but have not engaged with the approach. The counterfactual scenario is that No One Left Behind participants would not have been identified and reached by the approach, and therefore would have received no support. While there is a risk of selection bias, it is reduced by (i) the wide-ranging population covered by No One Left Behind across LEPs; (ii) the doubly robust DiD approach; and (iii) doing the impact analysis in changes and not levels. If any differences persist due to unobserved factors after robustness tests (e.g. switching values), theory-based approaches should be used.

In practice, this would involve using the DRT template to identify No One Left Behind participants in HMRC RTI and/or DWP RAPID datasets, using their NI number. To ensure the two groups are comparable, it will be key to balance them by assigning specific weights to variables that may affect No One Left Behind participation and outcomes. The key variables on which the estimation of weights would be based include:

• Demographic characteristics (e.g. age, sex, geographic location) in RAPID.
• Employment history (prior employment spells and earnings trajectories) in RAPID/RTI.
• Benefits history (type, duration and intensity of benefit receipt) in RAPID.
• Health-related indicators where feasible. These are available as proxies in RAPID (through receipt of health-related benefits, e.g. ESA, PIP), though these capture only the subset of individuals who have successfully applied for and been awarded these benefits.

6.1.4 Step 3: Quasi-experimental evaluation

Once the counterfactual and control groups have been established, the next step is to collate data on the outcomes of interest and conduct statistical analyses to quantify No One Left Behind’s impact. We believe that the most feasible statistical approach is a doubly robust DiD estimator. Under this approach, weights are constructed based on estimated propensity scores (the probability of being in the treatment group given observed characteristics), so that the weighted treatment and comparison groups are comparable at baseline on observed characteristics. Outcomes are then compared between the treatment and control groups using DiD. The doubly robust property means that the estimator produces consistent impact estimates if either the propensity score model or the outcome model is correctly specified. This means that only one of the two needs to be wrong for the estimator to recover the causal effect.

It is also worth noting that this approach enables a national evaluation with explicit recognition of local characteristics. This is achieved through a set of “local area control” variables, such as local unemployment rates, wages, sectoral distribution, urbanisation levels, and demographic characteristics. Together, these variables constitute a statistical description of specific areas or groups of areas that can then be used as weights in the inverse weighting process.

6.2 Economic evaluation

6.2.1 Step 4: Establishing methodological framework

The first step in the economic evaluation is to agree on the appropriate framework for assessing No One Left Behind’s VfM. Based on our evaluability assessment, we suggest conducting a Social Cost-Benefit Analysis that includes both the Exchequer (fiscal) and societal perspectives.

The framework must explicitly specify which benefits and costs should be included and how they differ between fiscal and societal perspectives. In summary, the fiscal benefits include: (i) reduction in social security payments; (ii) increase in the number of people paying taxes; (iii) increase in tax contributions from the existing tax base; (iv) increase in taxes due to an increase in lifetime earnings from improved education; (v) reduced NHS costs due to better health and wellbeing.

In contrast, the social benefits include: (i) an increase in economic output due to more people entering employment; (ii) an increase in economic output due to higher earnings; (iii) an increase in economic output due to higher lifetime earnings from better education; (iv) health or wellbeing benefits.

6.2.2 Step 5: Calculation of benefits

After all impacts have been identified and quantified, the evaluator can proceed with their monetisation. As previously mentioned, we expect health, wellbeing, and education to be evaluated using theory-based approaches. As a result, they will not be quantified or monetised. However, they will be discussed qualitatively alongside the monetised benefits.

In summary, we propose using a combination of published estimates and unit costs to monetise benefits. However, given that most of the identified sources are either UK-wide or England-based, appropriate adjustments are needed, as detailed in the Evaluation Challenges section, while a complete list of benefits is available in the Economic Evaluation section.

6.2.3 Step 6: Calculation of costs

The next step is to calculate the costs associated with No One Left Behind’s operations, over and above the selected counterfactual. We expect this to rely on the AIPs and the finance reports. However, it is worth noting that the national expenditure figures reported by the SG may not fully align with the actual funding received by Local Authorities for programme delivery. This discrepancy arises because the reported budget includes internal government costs that do not directly contribute to the provision of local services.

6.2.4 Step 7: Adjustments

The final step in the economic evaluation is to adjust the results to account for the inherent uncertainty in some estimates. In particular, the following adjustments need to be made:

• Deadweight: Any outcome that would have happened anyway in the "Business-as-Usual" scenario is considered deadweight and must be excluded from the programme's reported impact.
• Optimism bias: To achieve more accurate estimates, the Green Book recommends addressing optimism bias by making explicit adjustments at the beginning. This process involves increasing the estimated social costs and project durations while reducing the estimated social benefits by an adjustment factor.
• Social discounting: Social discounting is a method used to compare costs and benefits occurring at different times on a consistent basis. Health and wellbeing impacts should be discounted using the health and life discount rate, while all other impacts should be discounted on the STPR, as per Green Book guidance.
• Sensitivity & stress-testing: Evaluations often depend on assumptions that may not hold true, so sensitivity analysis is conducted to assess the reliability of the results. This process examines how variations in key assumptions - such as input costs, service demand, or the magnitude of social benefits - impact the proposal's summary metrics, like the Benefit-Cost Ratio.