Child poverty pathfinders in Dundee and Glasgow: phase two evaluation - report appendices

Appendix D – Technical details of break-even analysis for Dundee

Break-even analysis provides an estimate of the level of change in key outcomes that would be necessary for the programme’s benefits to match (or exceed) its costs. It is commonly used where clear evidence of the impacts of a program cannot be identified or the evaluation is taking place ex-ante before benefits will have been realised. This aligns with Cost Effectiveness Analysis in Green Book terms.

Break-even analysis can be used to create a ‘ready reckoner’ which compares the monetary costs of a programme to to the monetary value of the benefits they can be expected to produce, to help estimate the level of outcomes where it could be stated with reasonable confidence that there would be net positive in terms of the societal value produced on the outcomes included.

This can be used to inform future business cases (for similar interventions in other areas, for example), and plans for future monitoring systems, with a view to enabling more robust Cost Benefit Analysis in the future. This would require primary data to better evidence the number of people experiencing each outcome in the future, whether through monitoring data, surveys or administrative data.

This appendix provides further technical detail on the outcomes included in the break-even analysis included in Chapter 8, and on two outcomes considered but not ultimately presented there (but which may be useful for planning future business cases / monitoring of outcomes).

Outcome 1: New / improved sustainable employment

Outcome indicator: Job starts

Where a program leads to new employment, its contribution to the economy can be monetised in terms of its contribution to productivity in the economy. Where these jobs are a displacement of jobs that would have been created elsewhere in the country, the contribution to the local economy can be valued in terms of Gross Value Added (GVA) of that region to the Gross Domestic Product (GDP) of the country. We note that new job starts associated with the Dundee Pathfinder do not, in the main, represent additional jobs in the economy – the Pathfinder is helping people find jobs rather than creating jobs (as might be the case with an Intermediate Labour Market programme, for example). Jobs gained as a result of support from the Pathfinder would therefore represent a displacement of employment from one part of Dundee into the Pathfinder area. Given this, it can be argued that overall (at a macro Dundee or Scotland level) there are no direct additional economic or social benefits attributable to the pathfinders. However, if it can be argued that the Pathfinder channelled employment into an area of higher deprivation, then following HMT Green Book Guidance, it is possible to quantify the relative value of income for those on lower household incomes through welfare weighting (see Monetisation section below). For the purposes of this break-even analysis, we also assume that job starts entail moving someone from unemployment to full time employment for at least one year, and that the new job is living wage work.

The potential impacts on the exchequer are not estimated or included in the model. This is because these impacts would be classed by the Treasury as a transfer, whereby government money goes into the programme through the funding of the pathfinder and is saved through any impact on government savings (e.g. in benefits for those who obtain employment).

Monetization method

Economic value:

No new jobs were created by Pathfinder, meaning that all new job starts can be seen to represent a displacement of a job start elsewhere in Dundee. Therefore, the benefits are only additive to the local economy. However, given the lower average income areas of the area the Pathfinder focuses on, we can apply welfare weighting to account for the added benefits of generating new jobs in economically deprived areas. For this it is necessary to estimate first the economic value of a new job to the local economy, in terms of Gross Value Added (GVA) to the economy.

The most reliable estimates of the Gross Value Added to the local economy for new job starts comes from the What Works Centre^[1] which takes Government multiplier tables in HMT Green Book^[2] of 0.1 non-tradeable private sector jobs per new tradeable job created.

Employment effects into the economy can be translated into monetised value of employment through the multiplier effects from job creation in one sector on overall employment in a local or national economy. The additional benefit of the job comes from the increased demand for local goods and services produced when a new job-taker spends their salary in the economy, which contributes Gross Value Added to the local economy. We assume a living wage of £11.44 per hour for the UK in 2024. The average number of working hours in a year for a full-time employee is 2,080 hours = £23,796 * 0.1 as set out in the What Works Centre study = £2,380 economic benefit to the local economy per new job start.

However, as outlined above these employment effects are not additional, since no new jobs have been created. The additional value from the Pathfinder helping people in Linlathen and Mid Craigie find work is the welfare weighting to account for the lower incomes in the Dundee intervention area. The HMT Green Book using ONS data provides a welfare weight of 2.4 applied to the £2,380 GVA impact. This amounts to £5712 per new job start. It is the differential value of the welfare weight which can be applied to the new job start referrals facilitated by the Dundee Pathfinder, which amounts to £3,332.

Social (wellbeing value):

There is extensive evidence from the literature on subjective wellbeing through the life course that moving a person from unemployment to employment is statistically associated with an improvement in the life satisfaction of the individual, which is observed in large household datasets in the UK. This effect is observed even after controlling for income levels. This life satisfaction effect is observed in statistical models based on thousands of observations in the Understanding Society^[3] dataset (a large repeat (longitudinal) panel survey of around 100,000 GB residents) and is statistically significant. The effect size can be monetised following HMT Green Book guidance on the monetisation of subjective wellbeing using the WELLBY approach.

The Treasury Green Book Supplementary Wellbeing Guidance^[4] (2021) sets out the method for assigning monetary values to an increase in an individual’s self-reported wellbeing. The WELLBY method links a point improvement in life satisfaction to a change in QALY (Quality Adjusted Life Year) values, which is an established set of monetary values associated with quality of life, as used in Department of Health decision-making. This provides an annual monetary amount that is equivalent to a single point increase in life satisfaction (on a 0-10 scale).

Green Book guidance recommends that wellbeing values should only be applied where robust experimental or quasi-experimental analysis has been used to estimate impact. In this instance, we are using wellbeing values to provide indicative estimates of the benefits that could be provided by the CPP pilot in order to break-even to the costs incurred. Nonetheless, in the absence of robust impact assessment, we apply a number of conservative adjustments. First, we take the lower bound estimate from the Green Book Guidance (£10,000 per 1.0 unit increase in life satisfaction, uprated from 2019 to 2023-24 prices to £11,952). Second, we deal with additionality bias by applying a 40% deadweight and optimism bias to account for the counterfactual that some improvement in life satisfaction may have been observed without the Pathfinder intervention. This provides a more conservative break-even analysis. This also helps to avoid the potential criticism of over-attribution of value to CPP, to account for the fact that comparable employment outcomes could have been achieved without the Pathfinder intervention.

The wellbeing values of job starts are additive to the economic value of a new job, but again the total wellbeing benefit is arguably not directly attributable to the Pathfinder, since the job was already created and would have been filled with a candidate from elsewhere (who would have obtained that wellbeing value instead of someone supported by the Pathfinder). Therefore, the additional value comes from the differential welfare weighting – the fact that the job went to someone from a more deprived area.

In previous analysis reported in the HMT Green Book Supplementary Wellbeing Guidance^[5] using multivariate regressions on Understanding Society, analysis shows that having a full-time job is associated with +0. 46 higher average life satisfaction in comparison to being unemployed after controlling for (holding constant) a range of demographic factors that are known to influence wellbeing (see the end of this appendix for further details of the regression approach).

For break-even analysis, the HMT Green Book WELLBY value of £11,952 uprated to 2023-24) prices is multiplied by effect size of 0.46, which equates to £5,653per new job start per person. Applying the 40% deadweight, this equates to £3,391 of additional welfare weighted wellbeing value WELLBY value per job facilitated by the Dundee Pathfinder.

Exchequer savings:

Converse to the welfare weighting rationale above, if it can be argued that the Dundee Pathfinder was more likely to produce new job starts from an original position of unemployment, compared to a candidate who is already employed, then it could be argued that the Exchequer savings from reduced unemployment benefits would be greater for each new job start produced by the Pathfinder. However, without sufficient evidence to support such an assumption, we recommend not to apply Exchequer savings to the break-even analysis, as they can be considered to constitute a transfer of government funds from one part of government (program funding) to another (savings in government benefit payments). This may be something that could be revisited if the QED goes ahead and indicates that the Pathfinder has moved more people from unemployment to employment, compared with a control group.

Outcome 2: Improved financial stability

Outcome indicator: self-reported financial stability

A key aim of the PF is to help people become more financially resilient. Evidence shows that debt and financial insecurity exert a heavy mental toll on a person, and their family, which can be measured through its effects on subjective wellbeing. Where the pathfinder provides advice and/or directs a person towards financial or benefit support that may relieve financial debt and insecurity, it may be expected to improve the welfare of those beneficiaries. These benefits can be measured in terms of subjective wellbeing.

We take the Understanding Society question “How well would you say you yourself are managing financially these days?” and compare the wellbeing reported by those who say they are ‘Living comfortably’ or ‘Doing alright’, compared to those who are less financially comfortable (who respond they are ‘Just about getting by’, ‘Finding it quite difficult’, or ‘Finding it very difficult’.

By analysing the statistical association between life satisfaction and financial stability in a large national household survey, this enables us to generate a proxy of the improvement in life satisfaction that would be expected to arise if the CPP is able to make people feel more financially comfortable over the course of the year.

The average effect size of financial insecurity on wellbeing is estimated by running multivariate regressions in the Understanding Society dataset, to compare those who are in the upper end of the scale, to those in the bottom end of the scale, while controlling for (holding constant) other factors known to drive wellbeing. This quasi-experimental method gives an indicative wellbeing effect size which can be used for WELLBY valuation, with the necessary caveats that this is a nationally representative average within the Understanding dataset, and may therefore represent an over or under estimate compared to the specific characteristics of the Dundee CPP area. However, sample sizes in Understanding Society are too small for more localised analysis.

While there may be macro-economic impacts from improvements in financial stability and, in particular, reductions in household debt (leading to increased spending in the economy, with multiplier effects through the supply chain), it is not possible to quantify these in terms direct impacts to GDP. We therefore do not include economic benefits in the break-even analysis. Although there may be indirect Exchequer benefits associated with greater financial security, it is not possible to quantify this using any existing data or literature.

Monetisation method

Social value:

As set out above, for break-even analysis, we use a proxy question which is asked in the Understanding Society household survey, a large repeat (longitudinal) panel survey of around 100,000 GB residents. This provides a large sample on which to estimate the effects of managing financially on subjective wellbeing.

We take the Understanding Society question “How well would you say you yourself are managing financially these days?” and compare the wellbeing (life satisfaction, on the 0-7 scale question included in Understanding Society) reported by those who say they are ‘Living comfortably’ or ‘Doing alright’, compared to those who are less financially comfortable (who respond they are ‘Just about getting by’, ‘Finding it quite difficult’, or ‘Finding it very difficult’.

This provides a proxy of the improvement in life satisfaction that would be expected to arise if the CPP is able to make people feel more financially comfortable over the course of the year.

The analysis shows that ‘Living comfortably’ or ‘Doing alright’ is associated with high life satisfaction average of +0.388 on a scale of 1-7 life satisfaction points, compared to those who are ‘Just about getting by’, ‘Finding it quite difficult’, or ‘Finding it very difficult’ . Adjusting this to an 11 point scale and applying the WELLBY value £11,952 (2023-24) prices , this equates to £7,287 per year. Applying the 40% deadweight, this equates to a WELLBY value of £4,372 per person per year from a reduction in debt arrears associated with the improved financial security associated with the activities of the CPP in debt burden.

Outcome indicator 3: Improved mental health/wellbeing

Outcome indicator: improved mental wellbeing

The Dundee Pathfinder has explicitly taken a holistic, person-centred approach, that aims to help people improve not only their financial situation, but their wider wellbeing, both as an end in itself and as a key element in helping people achieve employability or financial goals, since poor mental health and wellbeing can be a barrier to these.

Monetisation method

Social value:

For break-even analysis, the HMT Green Book WELLBY value of £11,952 per unit increase (uprated to 2023-24 prices) can be used.

Therefore a significant improvement in mental wellbeing (equivalent to moving one point up a 11-point WELLBY scale) has a value of £11,952 per person per year. A smaller increase (0.5 points on an 11-point scale) would have a value of £5,976 per person per year.

What is not possible to establish, given the data available, is how many people are likely to have achieved wellbeing impacts at a level that would register as a 0.5 point increase in life satisfaction on an 11 point scale, or what intensity of support would be required to achieve this. For example, moving from unemployment to employment (which is a very significant change in one’s life circumstances) would be associated with around a 0.5 point improvement, as discussed above, so it might be argued that, for clients who do not move into employment, lower levels of change might be more likely. If before and after measures of life satisfaction were included in future monitoring data, this would enable better assessment of what level of improvement might be expected, and the WELLBY approach could be used to translate this into monetary values.

Alternative outcome indicators

As discussed in Chapter 8, two additional indicators were initially included in break-even modelling, but were deemed to be less relevant to the main aims of the Pathfinder. However, we include the analysis underpinning monetising these outcomes here for reference and potential use in any further business case for projects to which they are considered more relevant.

Health and wellbeing benefits from support with housing/heating

Outcome indicator:

Changes in indoor conditions due to expenditure changes (access to warm homes discount schemes and similar), translating these into health impacts using Quality Adjusted Life Years (QALYs) to enable monetary estimates of social value.

Monetisation method:

The aim of the Health Impact of Domestic Energy Efficiency Measures (HIDEEM) model developed by University College London is to quantify the indoor environmental conditions and monetise the health impact associated with energy efficiency changes in houses in Great Britain of the type and scale detailed in DECC's broad-ranging programme of interventions.” This provides an established exposure-response model methodology (Hamilton et al., 2015).^[6] The HIDEEM model is a building physics model that characterises the indoor environmental conditions of English houses for indoor winter temperatures, exposures to particle pollution, second hand tobacco smoke (STS), radon, mould growth and energy demand in relation to the energy performance of the dwelling. The model estimates the change in indoor conditions due to energy efficiency or changes in energy expenditure. These changes in conditions are used to estimate health impacts based on a combination of life table methods and directly modelled changes in disease prevalence (in this case relevant diseases are Cardiovascular (winter), Heart Attack, Stroke, Common mental disorder). More details can be found in Hamilton et al. (2015).

In the Warm Homes Discount Evaluation, The effect of providing an additional £140 of heating support for households of working age (ie those aged under 65) is computed to be an average increase of 0.18°C during wintertime conditions. The overall change in health resulting from the increase in predicted temperature is a benefit of 33,065 Quality Adjusted Life Years (QALY) over a 15-year period, with a mean benefit of 36.3 QALY/10,000 persons. The improvement in health assumes the temperature increase is maintained over the period. If monetised, these benefits could result in approximately £132 per capita of undiscounted health benefit over the 15-year period, assuming a value of £30,000 per QALY. Uprated to the 2024 QALY value of £70,000, the QALY benefits of the CPP referrals to energy support equivalent to the Warm Homes Discount Scheme would be £308 per capita of undiscounted health benefit over the 15-year period, equivalent to £20.53 per person year. Note, that this value it at the individual level, while the benefits of Warm Home Discount Scheme would be felt by the household. Using ONS figures on the average household size of 2.11 in Scotland, we can equivalise the QALY value of referrals to Warm Homes Discount schemes and equivalent to £43.36 per household per year. We do not apply a deadweight to this figure, as the counterfactual is already factored into the HIDEEM model.

Social connectedness: Sense of belonging in neighbourhood

Outcome indicator:

Social connectedness and sense of belonging in the local neighbourhood.

Monetisation method:

Analysis of the Understanding Society household panel dataset shows that those who report higher sense of belonging to their neighbourhood also report significantly higher levels of subjective wellbeing. Sense of belonging is measured in Understanding Society through the following question.

“Here are some statements about neighbourhoods. Please answer how strongly you agree or disagree with each statement.

I feel like I belong to this neighbourhood. (Agreement scale 1-5, where 1 is strongly agree and 5 is strongly disagree).

Social value:

Life satisfaction associated with sense of belonging to neighbourhood in Understanding Society has not been previously analysed. We applied the multivariate regression analysis method set out at the end of this Appendix to estimate the statistical relationship between those who agree or strongly agree with the sense of belonging statement and their significantly higher life satisfaction on average +0.147 on a scale of 1-7. Converted to a 11-point scale (in line with HMT Green Book Supplementary Guidance ) this equates to +0.231. Applying the WELLBY value of £11,952 in 2023-24 prices, this equates to £2,760.91. Applying the 40% deadweight, this equates to a WELLBY value of £1,656.55 per person per year from any improvement in confidence associated with the activities of the Pathfinder or similar intervention.

Summary of break-even analysis

Table D.2 shows the break-even analysis for the Dundee placement. The table reports only the lowest value outcomes, which gives a lower bound measure of the number of beneficiaries who need to be reached to break even, which accounts for some of the uncertainties inherent in predicting social value improvements in these outcomes in the local community.

As can be seen in the Table, with a combined economic benefit + WELLBY value of £6,723, job starts has the smallest break-even value with 78-96 individuals needing to be reached in order to break-even. Conversely, with a value of just £4,372 financial stability has the highest break-even value with 121-147 individuals needing to be reached in order to break-even.

Methodological note:

The application of the break-even calculator to a full business case would require data to be collected that evidences the number of beneficiaries who experienced these outcomes, either through primary data collection, whether as part of routine monitoring data or separate surveys of participants (e.g. by replicating the Understanding Society survey questions, on which this analysis was based, in a pre- and post-intervention survey on the target population) or administrative data (e.g. administrative records showing the number of participants who have achieved particular outcomes, like job starts, ideally with a control group so this can be more confidently attributed to the intervention).

Further detail on approach to Social Value Wellbeing Modelling and monetisation

The analysis underpinning the break-even tables presented above included the following steps/elements.

• Clean datasets and prepare Ordinary Least Square (OLS) regression analysis for outcome variables and accepted set of demographic controls for wellbeing analysis.
• Life satisfaction is the key dependent variable, for which monetisation estimates are available as recommended in the HMT Green Book Supplementary Guidance for Wellbeing for Appraisal. Wellbeing regressions apply the econometric model on large national datasets, such as the Understanding Society household panel survey using multivariate ordinary least squares (OLS) regression to the relevant dataset:

LS𝑖= 𝛼+𝛽1Outcomei+𝑿𝑖𝜷+𝜀𝑖"/>

Where LSi is life satisfaction for each respondent (i), the outcome (Outcomei) as recorded in the national dataset for each respondent, and Xi is a set of standard control variables to hold constant the sociodemographic and experiential factors that are known to drive subjective wellbeing.

• The standard control variables for OLS regressions on life satisfaction are: Log equivalised household income , Age and Age squared (to account for the U shaped relationship of age and wellbeing), Gender, Marital status, Education level, Employment status, resident in urban or rural area, Number of children, Government Office Region, Ownership of accommodation, Ethnicity, Physical health, Time of year that the interview took place, Survey wave. If available, Mental health and Religion and whether an individual is a carer variables (noting that in some datasets these questions may not be asked of a subset of the sample, leading to sample loss if included).
• Social benefits monetised in wellbeing: HMT Green Book Supplementary Guidance^[7] recommends that a one-point change in life satisfaction be converted to a monetary value by multiplying it by either the lower bound of £10,000 or upper bound of £16,000. We take the lower bound figure as this is the recommended standard value of one wellbeing adjusted life year – a one-point change in life satisfaction for one year – a ‘WELLBY’. The low value for a WELLBY is provided by an approach that pivots off the Green Book value of a QALY (Quality Adjusted Life Year). (£10,000 per 1.0 unit increase in life satisfaction, uprated from 2019 to 2023-24 prices to £11,952.

Note, all values are uprated to 2023-2024 Values: If based on pre-existing study, uprate wellbeing coefficient to current prices using (in this case 2023-24 financial year prices) using the GDP deflator (following HMT best practice) this equates £11,952. Note, monetisation from previous studies should not be directly transferred, as the method for monetising wellbeing impacts has been updated by the 2021 HMT Supplementary Guidance.

Calculations follow the principles set out in the HM Treasury Green Book which are set out below.

• Avoid double counting - Although some funded interventions are likely to meet multiple objectives it is important to choose one method only. This is because some methods will capture value from other outcomes. For example, supporting someone into work could reduce loneliness and improve financial health. However, because the benefits of reducing loneliness are calculated through improved subjective wellbeing, it is likely that benefits would be at least partially captured in the other outcomes. Therefore, simply adding the values in the table is likely to produce an overestimate.
• Account for deadweight – because the methods set out above do not include a counterfactual there is no way of knowing whether some of these outcomes would have happened any. For example, there may have been alternative activities in place at the area, local authority or Scotland level which could have also achieved these outcomes. Following guidance produced by the Homes & Communities Agency (2014) we suggest applying a deadweight of 40% to all estimates.