Fair Start Scotland: economic evaluation

Cost-benefit analysis

Timeframe

The scale of the benefits for any programme will depend on the length of time for which the benefits last. Indeed, per the aforementioned model parameter decisions, the assumption for how long the benefits last has the highest impact on model results. Therefore, it is important that the timeframe is chosen carefully based on the best available evidence. As mentioned, since the timeframe has such a significant impact on results, sensitivity checks will vary the length of persistence to show an optimistic scenario and a pessimistic scenario.

For the case of Fair Start Scotland, the persistence effect translates to the length of time in which participants have a job and earn a wage (after accounting for the counterfactual). This is not straightforward to measure for several reasons. The cohort that has been observed the longest (the 2018 cohort) has only been observed for three years (up to March 2021).^[10] This means that any benefits past the three-year mark are unobserved. Programmes like Fair Start Scotland may change the life trajectories of its participants, meaning that they will be more capable of finding and keeping a job throughout their entire life cycle.

As mentioned, the tracking period used for this analysis is three years after the start of the programme. It is clear that impacts will not cease immediately after the end of the tracking period, so it is reasonable to extrapolate estimated impacts into the future, especially for the 2019 and 2020 cohorts, who have a shorter period of observations available.

From the survey, it is possible to observe how long Fair Start Scotland participants keep their jobs for. Figure 2 shows how long the 2018 cohort stayed in work for between 2019 (Wave 1) and 2021 (Wave 3). In 2019, 25% of participants recorded having a job. By 2021, 26% participants were in work. From the survey data, it is evident that between 2019 (Wave 1) and 2021 (Wave 3), 9% of participants lost their job, but 10% gained a job. This means that the percentage of Fair Start Scotland participants who have a job is relatively constant (between 25% and 30%), but with some participants throughout the years either losing jobs or finding jobs, at a similar rate.^[11]

Source: Wave 3 survey of the 2018 cohort.

Additionally, Table 2 shows the observed average number of days that each cohort had worked for up to March 2021. Unsurprisingly, the average number of days with a job decreases for each cohort, as they are observed for a shorter period of time. Therefore, the 2018 cohort shows the fullest amount of information as it has been observed the longest.

Table 2. Average number of days that each cohort had worked for up to March 2021

Cohort	Length of time in which the cohort is observed	Average number of days with a job (up to March 2021)
2018 Cohort	Up to 3 years	1.5 years
2019 Cohort	Up to 2 years	298 days
2020 Cohort	Up to 1 year	110 days

Source: Wave 3 survey of the 2018, 2019, and 2020 cohort.

Focusing on the 2018 cohort, the conclusions that can be drawn from Figure 2 and Table 2 are that: (a) the percentage of individuals who have a job since the start of the programme is relatively stable over time, and (b) on average, individuals have worked for 1.5 years in the span of three years. Extrapolating this for the 2019 and 2020 cohort, the assumption for the central scenario is that the benefits last in total for 1.5 years.

Two main issues emerge with this assumption. Firstly, it assumes that for the 2018 cohort, the benefits do not last longer than the observed period of time. This is a very conservative assumption, as it is unlikely that those who reported having had a job for 2 years as of March 2021 will lose it immediately after the end of the survey. However, this conservative assumption will offset the fact that participants in the 2018 cohort who only had a job for a very short period of time are not observed and were not picked by the surveys.

Secondly, the 2019 and 2020 cohorts may have different outcomes to the 2018 cohort, especially because of the onset of the Covid-19 pandemic, which may result in less desirable outcomes when it comes to finding and keeping a job. However, there is emerging evidence from more recent management information that points to stronger preliminary outcomes for the 2019 and 2020 cohort. The improved preliminary outcomes are due to stronger collaboration between the Scottish Government and the providers, which may have offset any negative outcomes from the pandemic. This is discussed in more details in the sensitivity checks section.

All in all, a persistence effect of 1.5 years for all cohorts as a central scenario is relatively conservative, but it is the best observed estimate available given the evidence. For the sensitivity checks, scenarios where the benefits last for a shorter period of time (1 year) and for a longer period of time (2 years) are explored to calculate a realistic interval of the benefits and the benefit-cost ratio (BCR).

Results

By using the DWP SCBA model, estimates can be made on the benefits and costs from the perspective of participants, society, public finance, and employer. As mentioned, the focus will primarily be on the perspective of society as it provides the most nuanced picture, followed by the public finance perspective. Participants' perspective will also be shown.^[12]

As inputs, this model uses the programme costs per year, employment outcomes (hours worked, wage received, and number of days at work), as well as participant characteristics (age, marital status, disability status, and number of children). For participants in later cohorts, forecasts were used to provide estimates of any remaining employment outcomes and associated costs beyond the period for which data is held. This allows a comprehensive estimate of the full costs and benefits associated for the three cohorts.

The inputs used for cost data include service fees, implementation costs, non-supplier spend, and total outcome spend, both paid and outstanding. The costs per participant varied by year, so all costs were converted to real 2021/2022 pounds. This is why the programme cost will not add up exactly to the nominal cost provided by the Scottish Government. Total costs will also include the increase in travel costs and childcare costs, where applicable.

As discussed, 18 representative groups were used to show the range of participant characteristics and their Universal Credit treatment. Average employment outcomes were used for each of the groups to account for differences in wages and hours worked. The breakdown of the cost-benefit analysis for each of the groups is shown in Annex C.

The benefit-cost ratio is shown with and without the inclusion of QALYs and the distributional effect. The former measure helps compare the performance of Fair Start Scotland with the business case as well as the other programmes, while the latter shows a more accurate picture of the benefits accrued due to the programme. The accuracy of the results is dependent on the robustness of the impacts from which they are derived. A summary of the results is presented in Tables 3, 4, and 5 from society, participant, and public finance perspectives respectively.

The tables also include the results of the business case and the Work Programme for comparison. This is discussed in more detail in the next section.

Table 3. Results of the cost-benefit analysis from the perspective of society

Society's perspective	Business Case	Current CBA	Work Programme
Total benefits	£177.8m	£167.5m	-
Total costs	£117.3m	£82.8m	-
Programme costs	£109.2m	£69.9m
Financial BCR	1.5	2.0	2.7
Total benefits (including QALYs and redistributive effect)	-	£299.3m	-
Total BCR	-	3.6	-

Source: Analysis of management information, Wave 3survey data, cost data and post-2021 forecasts.

Table 4. Results of the cost-benefit analysis from the perspective of participants

Participants' perspective	Current CBA	Work Programme
Total benefits	£199.3m	-
Total costs	£140.4m	-
Financial BCR	1.4	1.3
Total benefits (including QALYs and redistributive effect)	£364.2m
Total BCR	2.6	-

Source: Analysis of management information, Wave 3 survey data, cost data and post-2021 forecasts.

Table 5. Results of the cost-benefit analysis from the perspective of public finance

Exchequer's perspective	Business Case	Current CBA	Work Programme
Total benefits	£128.7m	£110.7m	-
Programme Costs	£109.2m	£69.9m	-
Total BCR	1.2	1.6	2.4

Source: Analysis of management information, Wave 3survey data, cost data and post-2021 forecasts.

The results show that the programme is a net positive to society; for each £1 spent on the programme, the estimated benefit is £2 in 'financial' terms or £3.60 in terms of overall benefit to society. The programme is also a net positive in terms of public finance, with every £1 spent yielding £1.60 in benefits to public finance. The programme is also a net positive from the participants' perspective; for each £1 spent, the estimated benefit is £1.40 in 'financial' terms or £2.60 when QALYs and redistribution are considered.

Sensitivity checks

As noted in the methodology section, there is considerable uncertainty regarding the accuracy of the baseline assumptions. For this reason, the cost-benefit analysis of Fair Start Scotland was estimated on the basis of several potential scenarios. These scenarios are defined by varying: (a) the length of time for which benefits persist, and (b) the counterfactual – how many of the participants would have had a job in the absence of Fair Start Scotland.

The rationale for varying the persistence effect is that the central scenario is based on an estimation of how long the benefits last for rather than a true, observable figure. Therefore, an assumption is made that, for all 3 cohorts, the benefits only last for one year.

Another assumption is made that the benefits of the programme last for 2 years. This takes into consideration the potential need to apply an 'uplift' for the 2019 and 2020 cohort, as they performed better than the 2018 cohort for the time they were observed. Additionally, some findings from the literature indicate that programmes like Fair Start Scotland can have a long lasting but diminishing effect over the lifecycle that cumulatively adds up to 2 years.^[13]

As for the rationale for varying the counterfactual, it could be as low as 0% especially at the onset of the Covid-19 pandemic, where it is extremely unlikely that those farther away from the labour market would be able to get a job in the absence of Fair Start Scotland. However, it could also be as high as 40% due to the composition of participants that ended up joining Fair Start Scotland being overrepresented among the Core group and less so among the Advanced and Intense groups.

Given these different scenarios, the results of the sensitivity checks are shown in Table 6. The first two columns correspond to the analysis from society's perspective, while the third refers to the public finance perspective. A pessimistic and an optimistic scenario are also defined; where the first combines shorter persistence and a higher counterfactual, while the later combines a longer persistence and a lower counterfactual.

Table 6. Results of the different scenarios of the sensitivity checks from the perspective of society

Scenarios	Financial BCR (society's perspective)	Total BCR (society's perspective)	Public Finance BCR
Central scenario	2.0	3.6	1.6
1-year persistent effect (observed)	1.5	2.7	1.1
2 years persistent effect	2.6	4.6	2.1
40% counterfactual	1.6	2.8	1.2
0% counterfactual	2.4	4.3	2.0
Pessimistic scenario	1.1	2.0	0.8
Optimistic scenario	3.1	5.5	2.7

Source: Analysis of management information, Wave 3survey data, cost data and post-2021 forecasts.

The sensitivity checks demonstrate that, even in the most pessimistic scenario and the narrowest assessment of net benefits, the programme more than breaks even, with every £1 in cost resulting in £1.10 in 'financial' benefits to society. The same is not true from the point of view of public finance, as £1 in costs yields only £0.80 in benefits. While increasing the counterfactual does significantly impact the results, the most substantial change comes from shortening the amount of time that benefits last for. It is important to reiterate here that it is extremely unlikely that the effects will only persist for 1 year, as this assumes that Fair Start Scotland participants who have had jobs for years will lose them immediately after March 2021, the last date on which they are observed.

In the most optimistic scenario, Fair Start Scotland yields £3.10 in financial benefits to society or £5.50 in total benefits for each £1 spent. Again, this is more a result of increasing the length of time the benefits last for rather than a result of decreasing the counterfactual. Whether a 2-year persistence effect is realistic or not is hard to tell given the available data. The central estimate is still relatively conservative, however, given the strict cut off at 1.5 years for the 2018 cohort.

Other measures of performance

The BCR gives the best measure of value for money for an evaluation as it considers a wide range of impacts of a programme. One drawback of the BCR, however, is that it may not always be available for comparison with other programmes, as it is relatively complex to calculate. Therefore, the purpose of this section is to show more straightforward value for money measures – namely cost per outcome – that can be more easily compared across different programmes.

Interpreting and comparing these cost effectiveness measures of Fair Start Scotland with other programmes should be done carefully. As mentioned earlier, while this service did not draw as many participants or achieve as many job outcomes as anticipated, it still performed well because of the of the stability of jobs achieved. Cost effectiveness measures are only able to capture quantity relative to cost which, for the case of Fair Start Scotland, does not give the full picture. Table 7 shows the cost per job start across participant groups and Lots. The cost per job start is calculated based on information for the 2018 cohort only. This is because the job starts of the participants, as well as programme costs from the 2018 cohort have been most fully observed by the time this analysis has been undertaken. Consequently, the advantage of calculating the cost per job start for the 2018 cohort only is that it provides the most accurate representation of the actual costs and outcomes that took place. On the other hand, it should be taken into account that the fact that only the first cohort of the programme is studied may overestimate the total cost per job start, as we expect costs to be higher at the beginning of a programme (e.g. due to start-up costs). More information about the methodology for calculating the cost per job start across participant group and Lot is detailed in Annex B.^[14]

Table 7. Cost per job start for Year 1 across participant group and Lot

Participant group	Cost per job start
Core	£4,849
Advanced	£8,516
Intense	£10,261
Lot	Cost per job start
Glasgow	£6,586
Lanarkshire	£5,469
Tayside	£5,461
Forth Valley	£8,897
East	£6,005
South West	£7,189
North East	£9,129
Highlands and Islands	£11,442
West	£8,886
Total	£6,754

Source: Analysis of management information and cost data for the 2018 cohort.

On average, the cost per job start is £6,754. However, there is a significant difference in cost across characteristics. Unsurprisingly, the cost per job start for participant group is lower for the Core group than Advanced or Intense. This is because, as mentioned, the providers are paid at a lower rate for the Core group than the other two. Finally, some Lots had a much higher cost per job start than others, going as low as £5,461 in Tayside and as high as £11,442 in the Highlands and Islands. The reason for this discrepancy is discussed in the chapter on inclusive growth.