Benchmarking Fair Start Scotland's performance
In order to put the Fair Start Scotland evaluation in context, the results of the analysis will be compared to the business case and other similar programmes. The former will allow for assessing the actual performance of Fair Start Scotland compared to the expected performance, while the latter will help benchmark the experience of Fair Start Scotland against comparable programmes.
Comparison with the business case
Overall, as illustrated in Tables 3 and 5, Fair Start Scotland performed much better than expected compared to the business case. This is mainly due to the lower implementation cost of Fair Start Scotland than initially expected, as the benefits are remarkably similar ex-post and ex-ante. There are several differences between the expected and actual performance of Fair Start Scotland. Some of these are shown in Table 8.
|Indicator||Business case||Actual performance|
|Share of referrals that turned into participants and started on the service||70%||67%|
|Number of participants that started on the service||38,000||32,505|
|Number of participants that started on the service per year||12,642 per year||10,063 (2018)|
|Percentage of Core group participants||14%||41%|
|Percentage of Advanced group participants||50%||33%|
|Percentage of Intense group participants||36%||24%|
|Percentage of participants who achieved the 13-week outcome||36%||23% (for the 2018 cohort)|
|Percentage of participants who achieved the 26-week outcome||30%||18% (for the 2018 cohort)|
|Percentage of participants who achieved the 52-week outcome||25%||14% (for the 2018 cohort)|
|Programme cost per start||£2,928||£2,181|
Source: Business case, management information data, and performance data.
Fair Start Scotland had lower overall participants starting on the service compared to the business case (38K vs 32.5K). It also had a much higher representation among the Core group in the actual performance than the business case. Subsequently, it had lower participation in the Advanced (33% compared to 50% in the business case) and Intense groups than anticipated (24% compared to 36% in the business case). Finally, fewer participants achieved the 13-, 26-, and 52-weeks outcomes (for the 2018 cohort) than anticipated in the business case.
These discrepancies can explain why costs were lower while benefits were higher when compared to the business case. This is best illustrated when comparing the management information to the survey results regarding how long the benefits last. For the 2018 cohort, the share of people who achieved the 52 weeks outcome is 14%. But when people were surveyed (up to) three years later, 25% reported that they had a job. This provides evidence of job outcomes being achieved but without triggering incentive payment to providers. Therefore, despite perceived underperformance compared to business case expectations, evidence suggests that Fair Start Scotland may have achieved higher benefits once long-term benefits are considered.
An additional reason for the discrepancy between the actual and anticipated performance is the reduced costs due to the higher share of Core participants. Providers are paid less for Core participants than for the other two groups, resulting in similar benefits but lower costs. This has an implication for the inclusive growth aspect of the programme, as it was less able to target individuals with high barriers to labour market entry than it set out to do. This will be discussed in more detail in the inclusive growth section.
Comparison with other programmes
It is important to be very careful when comparing the Fair Start Scotland evaluation results with those of other employment programmes. Even if programmes have similar overarching goals, they may still differ in their specific aims, design, and reach. The way outcomes are defined and measured is likely to be heterogeneous. The scale of the programme can also distort results; for example, bigger, nation-wide programmes that operate for longer are likely to benefit from economies of scale.
In addition to specific programme characteristics, evaluation design and context may also hinder direct comparisons. For example, other programme evaluations are likely to include or exclude different aspects of cost-benefit analysis compared to the choices made in this evaluation. Some programmes may use control groups to assess the counterfactual or may follow participants for a longer period of time. Additionally, in many cases, the relevant information is not shared clearly and transparently, making it unclear whether comparisons are on a like-for-like basis.
Nevertheless, it is important to benchmark Fair Start Scotland against similar services to understand how well it performed relative to others. This section compares the performance of Fair Start Scotland to other programmes on several measures, including value for money, performance, reach, costs, and job outcomes achieved. The programmes used for comparison are either UK-wide or Scotland based. The reason international comparisons are not undertaken is to ensure that the context is as similar as possible.
In terms of cost-benefit analysis, the most suitable comparator to Fair Start Scotland identified in the literature is the Work Programme. This is mainly because the evaluation of the Work Programme uses the DWP SCBA, meaning BCR comparisons are relatively straightforward. The Work Programme also aimed to move people into employment and has a payment-by-results model similar to Fair Start Scotland.
However, there are also many key differences. The Work Programme was a very large, UK-wide programme that operated for 6 years with close to 2 million participants. Unlike Fair Start Scotland, the Work Programme was not voluntary and did not target people with particular characteristics. These differences mean that comparisons need to be interpreted carefully, and any conclusions made need to be caveated within this context. Table 9 shows a comparison between the different BCRs of Fair Start Scotland and the Work Programme.
|Comparison measures||Society 'financial' BCR||Public Finance BCR||Participant BCR|
|Fair Start Scotland||2.0||1.6||1.4|
Source: Analysis of management information, Wave 3 survey data, cost data, and the Work Programme's quantitative impact assessment.
The results are relatively similar, with Fair Start Scotland performing slightly less well than the Work Programme in terms of BCR measures of value for money. This is likely due to the differences in design and scope, as discussed previously. This includes distance and the size of the population. Having to work in remote and scarcely populated areas of Scotland may mean that employment provision is more difficult and costly when compared to the Work Programme. Additionally, Fair Start Scotland is specifically designed to work with participants who have high barriers to finding a job and are further from the labour market. This means that the cost per participant is likely to be much higher than that of the Work Programme. Finally, as previously mentioned, the Work Programme was a very large programme that lasted 6 years and included more than 2 million participants. This will have led to economies of scale that are likely to affect value for money positively.
The voluntary nature of Fair Start Scotland may also have implications for the results. Unlike the Work Programme, participation in Fair Start Scotland is not a condition for receiving or continuing to receive benefits. This may have several effects. On the one hand, this means participants may have less incentive to remain in the programme until they find employment, potentially implying higher incurred costs for Fair Start Scotland that are not offset by benefits. However, because participants voluntarily decide whether they want to participate in Fair Start Scotland or not, the programme likely attracts those who are genuinely motivated and want to find employment. This means that, compared to the Work Programme, Fair Start Scotland is likely to help fewer people achieve job outcomes but those who do achieve better and longer-lasting outcomes.
Beyond BCR measures, there is some scope for comparison between Fair Start Scotland and the Work Programme regarding participants' experience. Participants with experience in the Work Programme felt that Fair Start Scotland key workers are more supportive and respectful than Work Programme advisers. Participants also disliked the compulsory nature of the Work Programme and the risk of sanctions for non-completion or non-participation.
It is also possible to benchmark how well Fair Start Scotland performed relative to other programmes by using alternative measures of performance. Given the design and the available information on performance and reach, the Work and Health Programme and the Work Choice Programme are identified as suitable comparators. The two programmes share some similarities with Fair Start Scotland. All three aim to help people find and stay in employment. They also target similar (but not identical) groups, namely people with disabilities and those farther away from the labour market. All three programmes are, for the most part, voluntary.
Table 10 compares Fair Start Scotland with the Work and Health Programme and the Work Choice Programme on reach. Other information about the programmes is also shown, including years and location of operation and target population. This is to ensure that the context is taken into account when the comparison is undertaken.
|Comparison measures||Number of starts||Number of years in operation||Average number of starts per year||Target population||Operation locations|
|Work Choice Programme||158,440||2010/11 -2017/18 (8 years)||19,805||People with disabilities||UK wide|
|Work and Health Programme||150,104||2017-2021 (3.5 year)||42,887||People with disabilities, long term unemployed, early access||England and Wales|
|Fair Start Scotland||32,505||2018/19-2020/21 (3 years)||10,835||As per definitions in Table 1||Scotland|
Given the significantly narrower operation of Fair Start Scotland compared to the other two programmes, it performed remarkably well in terms of reach. While both the Work and Health Programme and the Work Choice Programme reached a larger number of people per year, they operated at a much larger scale than Fair Start Scotland.
In order to draw firm conclusions about Fair Start Scotland's reach relative to the other programmes, performance should be benchmarked against population measures. However, given the diversity of groups, locations, and years, this is not straightforward and needs to be interpreted carefully. Population measures are illustrated in Table 11.
|Comparison measures||Average annual number of starts||Average number of unemployed people in the region/time period of operation||Percentage of starts against unemployed population|
|Work and Health Programme (2017-2021)||42.9K||1.3m||3%|
|Fair Start Scotland (2018/19-2020/21)||10.8K||110K||10%|
|Comparison measures||Average annual number of starts with disabilities||Average number of unemployed people with disabilities in the region/time period of operation||Percentage of starts with disabilities against unemployed population with disabilities|
|Work Choice Programme (2013-2018)||18.2K||404K||5%|
|Work and Health Programme (2017-2021)||32K||351K||9%|
|Fair Start Scotland (2018/19-2020/21)||4.7K||29.5K||16%|
Source: Performance data of the Work Choice Programme, Work and Health Programme, and Fair Start Scotland and national statistics on the number of unemployed people.
Table 11 shows that the reach of Fair Start Scotland was higher than the Work and Health Programme. However, these results need to be interpreted carefully as the population figures are imperfect. Additionally, there may be some inconsistencies across definitions of the targeted population across the years, locations, and programmes. Suffice to say that Fair Start Scotland performed comparatively well in terms of reach.
It is also possible to compare the performance of Fair Start Scotland to the other two programmes in terms of job outcomes achieved. It is important to note that each one of the programmes defines job outcomes very differently, as noted in the table, so any conclusions made need to be considered carefully.
|Comparison measures||Referral to start rate||Job outcome 1||Job outcome 2||Job outcome 3|
|Work and Health (2017-2021)||62%||5% (Reached a level of earnings once in employment within 6 months)||14% (Reached a level of earnings once in employment within 12 months)||-|
|Work Choice Programme (2010-2017)||75%||34% (Job is sustained for 3 months in a 12-month period)||21% (Job is sustained for 6 months in a 24-month period)||-|
|Fair Start Scotland (2018)||67%||23% (In work for 13 weeks in a 16-week period)||18% (In work for 26 weeks in a 30-week period)||14% (In work for 52 weeks in a 60-week period)|
Source: Performance data of the Work Choice Programme, Work and Health Programme, and Fair Start Scotland.
Fair Start Scotland performed relatively well compared to the other two programmes. In terms of retention, Fair Start Scotland's referral-to-start rate was between that of Work Choice and Work and Health. Job outcomes for Fair Start Scotland and the Work Choice Programme are more comparable than the Work and Health Programme because the latter takes into account the level of earning, not just the longevity of the job outcome. Fair Start Scotland achieved a lower rate of job outcome 1 than the Work Choice Programme. This is likely due to the time needed for the job outcomes to be achieved, which is much tighter for Fair Start Scotland (16 weeks) than the Work Choice Programme (12 months). The rates of achieving job outcome 2 confirm the evidence that Fair Start Scotland leads to stable jobs; 78% (or 18 out of 23) of Fair Start participants who achieved the first job outcome went on to achieve the second one, as opposed to 62% (21 out of 34) for the Work Choice Programme.
Finally, it is possible to compare Fair Start Scotland with other programmes in terms of cost per job outcome. Similar programmes identified in the literature that use these measures are the Working for Families Fund and New Futures Fund. Table 13 compares the cost per participant and job outcome across the three programmes.
|Comparison measures||Cost per start/participant||Cost per job outcome 1||Cost per job outcome 2||Cost per job outcome 3|
|Working for Families Fund (2004-2008)||£1,642 nominal £2,430 real||£3,382 nominal £5,004 real||-||-|
|New Futures Fund (1998-2005)||£2,100 nominal £3,395 real||£6,100 nominal £9,863 real||£9,300 nominal £15,037 real||£21,100 nominal £34,116 real|
|Fair Start Scotland (2018-2021)||£2,181 real||£9,918 real (£5,630 real per job start)||£13,928 real||£24,265 real|
The two programmes are suitable comparators to Fair Start because all three operated in Scotland and targeted people with similar profiles. However, since the other two programmes are significantly older than Fair Start Scotland, it is important to use real costs when making the comparison. Additionally, job outcomes are defined very differently in the three programmes. As mentioned, Fair Start Scotland defined job outcomes 1, 2, and 3 as achieving 13, 26, and 52 weeks in employment, respectively. The Working for Families Fund defined job outcomes as the transition into employment. The New Futures Fund defined job outcomes 1, 2, and 3 as moving into education, supported employment, and employment, respectively.
Taking all of this into account, Fair Start Scotland had lower costs per start but a higher cost per job outcome than the other two programmes. This is not surprising, given Fair Start Scotland's much stricter definition of job outcome. Indeed, accounting for differences in job outcome definitions, it may be fairer to compare the cost per job outcome 1 of Working for Families Fund (£5,004) to the cost per job start of Fair Start Scotland (£5,630). Considering this, Fair Start Scotland performed remarkably well in comparison to both programmes.