Scotland's Devolved Employment Services: Review of Employment Progression Methodology

1. How do we collect our data?

The No One Left Behind data included in our quarterly publication is provided to us by the 32 Local Authorities (the data providers). They each submit a quarterly data template, which we then merge and analyse. We have used three separate data collection templates since the start of No One Left Behind:

• Year 1 data collection exercise: We conducted a separate, one-off exercise to collect individual level data on year 1 participants. The data from this exercise is combined with data collected for subsequent periods and closely aligns with the ‘old’ data template. 
• ‘Old’ No One Left Behind Data Template (April 2020 – September 2022): From April 2020 up to September 2022, we used the ‘old’ data template to collect individual level data each quarter. Most of the variables collected in this template are retained in the newest template, though in many cases these have been expanded on.
• SMF Data Template (October 2022 – Present): From October 2022 onwards, we sent out the ‘new’ data template to collect individual level data each quarter. This template is a collaboration between Scottish and Local Government and partners, and it aligns with the Employability Shared Measurement Framework (Opens in a new window) themes of “reach” and “progression”. It should be noted that 11 Local Authorities transitioned to this new template from July 2022.

As the progression data we publish is a combination of these three data sources, we need to consider the differences between the collections when we consider improvements. For example, it is possible to report on the number of participants entering different positive destinations such as employment throughout the full period of No One Left Behind. However, there are differences in the employment follow-up checks and questions, which impacts how we present this data. Please see Appendix A for a full comparison of both templates.

2. What data do we currently publish?

We currently publish (pre-May 2025) three sets of progression tables: 

1. Progression by demographic variables including sex, age group, ethnicity, disability status and parental status;
2. Progression by long term health condition;
3. Progression by Local Authority.

These tables show the total count of those who entered employment from April 2019 onwards. There are no breakdowns by year and quarter, and we do not distinguish between participants who just started receiving support and participants who have been receiving support for some time.

The tables also show our follow-up variables. This includes the number of participants who entered employment and are still in employment at 4, 13, 26 and 52-week follow-up points. A separate table shows this information as a percentage of all participants who entered employment. For example, if 1,000 participants entered employment and 200 are still found to be in employment at the 52-week follow-up point, this table will show 20%.

Participants who ‘entered employment’ are also broken down into those who were supported by a subsidy to do so (this includes Employer Recruitment Incentives and inwork training allowances), those who started a modern apprenticeship, and those who entered self-employment. 

Our progression tables also include information on those who entered Further/Higher Education and/or training, those who gained a qualification, and those who achieved other outcomes including work experience, re-engagement with school and volunteering. A summary of what is included in our current progression tables can be found in the table below.

Table 1: Summary of our currently published progression tables (pre-May 2025)

Outcome achieved	Further breakdowns	Time period	Percentages
Entered employment	Supported by a subsidy; Started modern apprenticeship; Entered self-employment; In employment after 4 weeks; In employment after 13 weeks; In employment after 26 weeks; In employment after 52 weeks.	The whole duration of No One Left Behind: April 2019 – most recently published quarter	Numbers as a percentage of all participants who started to receive No One Left Behind support; Numbers as a percentage of all participants who entered employment.
Entered FE/HE and/or training	FE/HE training; LTU accredited training.	The whole duration of No One Left Behind: April 2019 – most recently published quarter	Numbers as a percentage of all participants who started to receive No One Left Behind support.
Gained a qualification	FE/HE qualifications; Accredited training qualifications.	The whole duration of No One Left Behind: April 2019 – most recently published quarter	Numbers as a percentage of all participants who started to receive No One Left Behind support.
Achieved other outcomes	Started work experience; Re-engaged with school; Started volunteering; Supported by a subsidy.	The whole duration of No One Left Behind: April 2019 – most recently published quarter	Numbers as a percentage of all participants who started to receive No One Left Behind support.

3. What are the challenges with our current approach, and what are the solutions?

Although our currently published progression tables do provide insight into the progression journeys of No One Left Behind participants, this approach has certain limitations. We conducted a deep dive of the data and methodology, in which we examined these limitations and explored solutions. This section sets out the results of this exercise.  

 

1. Our tables do not indicate when a participant started, and how long it takes for them to reach a certain outcome.

The currently published progression tables do not break down information by year and quarter. This contributes to misconceptions about No One Left Behind. For example, as of June 2024, 31% of No One Left Behind participants had entered employment. We know, however, that most participants need time being supported before they are ready to enter employment. By only showing the number of participants who entered employment as a percentage of all participants, including those who started in the most recent quarters, this percentage is not necessarily a fair reflection. Showing breakdowns by the year and quarter that participants started would therefore provide a more nuanced and impactful picture of employability support and progression.

Year and quarter breakdowns would also allow us to calculate and show the average amount of time participants receive support before they enter employment. Showing this information would provide an indication of the time spent by employability partners to support participants into employment.

As part of our deep dive, we examined whether we could implement these changes in accordance with the data quality standards set out by the OSR. In terms of counting the number of participants that are entering employment, we found no major concerns with either the data returned by data suppliers or the approach we use for our calculations. We also found that all participants have a valid start date, and a valid employment start date. This is true for the data returned as part of the old template as well as the new template.

This means that we can confidently split progression by participant start year and quarter and calculate the average amount of time it takes for a participant to reach employment after starting on No One Left Behind. We will not need to collect any new or additional data to do this.

2. For follow-up variables, we do not currently factor whether a participant has been in employment long enough to reach the follow-up point.

In the previous section, we described how we calculate the number of participants still in employment at the various follow-up points as a percentage of all participants who entered employment. The example we used showed that, if 1,000 participants reached employment as of 31 July 2024, and 200 of these were still found to be in employment at the 52-week follow-up, our tables state that 20% of participants who entered employment are still in employment at the 52-week follow-up point. 

This approach fails to consider that not every participant who has started employment will have been employed long enough to reach the 52-week follow-up point and is therefore an underestimate of the true rate. Removing participants who have not been employed long enough to reach this follow-up point would better demonstrate employment progression. Employment progression is calculated as follows: key workers contact participants at set follow-up intervals and record their status. There are different approaches used for contacting participants, for example phone calls, text messages etc and this can vary by locality. Participants do not need to have remained in employment for the full follow-up period. They could enter employment, leave, then have returned to employment by the follow-up and be counted. To build on the previous example, if of the 1,000 participants who started employment only 600 started employment 52 weeks before the end of the reporting period, the percentage of participants found to be in employment at 52 weeks would be 200/600, or 33%, rather than 200/1,000, or 20%.

Given all participants have a valid employment start date, calculating whether a participant has had enough time to reach the follow-up point can be implemented immediately.

 

3. For the follow-up variables, we only consider the number of participants who are still in employment and do not distinguish between other responses.

For the follow-up variables, we currently only consider the number and percentage of participants who are still in employment at the follow-up point. Both the old template and the new template, however, include multiple response categories besides “In

employment”, including: “In Further Education/Higher Education”; “In Accredited Training”; “In school”; “Not in employment or other destination listed”; “Inactive”; “Unknown; N/A – no engagement”; “Not recorded”. 

Presenting statistics on these response categories would allow us to distinguish between those who are not in employment, those who are in other positive destinations, and those who could not be reached. The latter is especially important, given that data suppliers have indicated challenges contacting participants once they stop receiving support. Including these statistics would therefore provide a more nuanced depiction of participant outcomes.

However, as part of our deep dive into the follow-up responses, we identified data quality issues. We noted that while participants entering employment in the old template were being counted, follow-up information for these participants was not included if only collected in the old template. Because of the time between participants starting support, entering employment and the full follow-up period of 52-weeks, many participants who started on the old template had (part of) their progression journeys recorded in the new template. However, there are still a number of participants whose entire progression journey was recorded within the old template. That follow-up information has previously not been included in our statistics. This was artificially lowering the percentage of participants still in employment for each of the follow-up points and has now been corrected and will show from May 2025.

We also found that, while “In employment” values have been recorded consistently, the number of unknowns/blanks is very high and “Not in employment” values are often not recorded at all. This is true for the data collected within the new template as well as the old template. Analysis and further breakdowns of the issues we encountered can be found in Appendix B.

This means that, although still considered a high priority for improving the value of these statistics, including follow-up responses other than “In employment” will require additional work with data suppliers. We need to understand why this information is currently not returned at the same quality as other information, what changes could be considered to improve on the quality of this information and any difficulties with implementing these changes. We will also need to consider any difficulties around improving historical data and implications for the presentation of these statistics. This will require additional work and will be considered alongside other improvements we will make to the data. Work in this area has started as we issued a survey to data suppliers to better understand their use of these other follow-up options.

Overall, it should be noted that this means participants still in employment but who do not respond to follow-ups are not included in the numerator but are still included in the denominator and therefore we may still be underestimating the proportion of those still in employment.

 

4. We do not report additional outcomes if the participant has previously entered employment.

Within No One Left Behind, it is possible to report on more than one outcome for participants under certain circumstances. If a participant achieves another outcome outside of the 52-week follow-up period, a new row of data with the additional outcome information (including follow-ups) should be added. If a participant achieves the same outcome type within a 52-week follow-up period (for example, they enter one job then enter another), a new row does not need to be added. This includes different employment outcomes such as moving from employed to self-employed.

Currently, additional outcomes are only counted for participants entering employment if their first outcome was not entered employment. This ensures that each participant is only counted once. Our statistics, therefore, are a count of the unique number of participants that have entered employment and not a count of the number of times the entered employment outcome has been achieved. For example, if 100 participants entered employment and then returned for No One Left Behind support outside their original 52week follow-up period, they may enter employment a second time and this data would be returned to us as an additional row. However, we would not currently include this as part of our statistics.

The inclusion of additional outcomes would require consideration and development in terms of presentation of the statistics. Currently, it is straightforward to say X number of people entered employment. However, including additional outcomes would change this. As the number of No One Left Behind participants continues to increase, the likelihood some participants will return increases and therefore additional outcomes may become a more pressing consideration. It should be noted there is a precedence in how this might be managed by referring to Fair Start Scotland after re-starts were introduced.

 

5. While detailed employment information is collected, we do not currently publish this data.

For participants who enter employment, we also collect information on contract type, number of hours worked per week, rate of pay and employment sector. The SMF template also includes follow-up points for these variables. We currently do not publish this information.

Because this information was only collected as part of the new template, we can only report this information for participants who entered employment after October 2022. We have not reported this information to date and initial quality checks have raised concerns around the quality of this data. Before we can start publishing this, we would therefore first need to do a deep dive into the quality of this data and determine the work required to improve it enough to meet the standards for a statistical publication.

4. What are the changes proposed to take place from May 2025 onwards?

As explained in the previous section, there are two changes we can implement with immediate effect:

1. Break down progression statistics by start year and quarter and calculate the average amount of time it takes for participants to reach employment after starting to receive No One Left Behind support
2. Calculate the number of participants who have had enough time to reach the follow-up point and use that as the denominator to calculate employment progression rates.

To do this, we do not need to change the way participants are counted as entering employment as this was found to work effectively. We did, however, encounter a coding issue where follow-up responses were not included if exclusively recorded in the old template. This issue has now been rectified, and follow-up responses recorded exclusively in the old template will be included going forward. 

Going forward, the denominator used to calculate employment progression rates will only include participants if they have been in employment long enough to have reached that follow-up point, based on their employment start date. For example, we will only include participants who started employment at least 13 weeks prior to the end of the reporting period in the denominator for calculating rates for 13-week follow-ups. The table below shows how this changes the denominator for each follow-up, using data from our October 2024 publication.

Table 2: Denominator for each follow-up using old and new approach

Denominator	4 weeks	13 weeks	26 weeks	52 weeks
Old	20,743	20,743	20,743	20,743
New	18,402	19,524	18,149	14,880

In our old methodology, the denominator included all participants who entered employment (20,743 as per our October 2024 publication). By removing those who have not been in employment long enough to reach the follow-up point, the new denominator is consistently lower than the old denominator, as would be expected. It should be noted that the 4-week follow-up was a new variable added to the SMF Data reporting template and is therefore not available for any data collected on the old template, meaning this information is unavailable for some participants. The result of this can be observed in table 3 which shows less participants reached the 4-week check-in point than reached 13-weeks. 

Using these new denominators will provide a better reflection of the percentage of participants that remain in employment at different follow-up points, as can be seen in the table below.

Table 3: Comparison of numbers and percentages using old and new methodology

Method	4-week follow-up	13-week follow-up	26-week follow-up	52-week follow-up
Old Entered Employment	8,657	8,130	7,102	4,760
New Entered Employment	8,603	11,247	9,419	4,943
Old Progression Rate	41.7%	39.2%	34.2%	20.4%
New Progression Rate	46.8%	57.6%	51.9%	33.2%

Please also note the following two corrections we have made to previous data following this deep dive:

1. The ‘new entered employment’ numbers have increased for the 13, 26 and 52 week follow-up points compared to the ‘old entered employment’ numbers as we have now corrected the previously described coding issue meaning participants from the old template who were previously missed are now included.
2. As 4-week follow-up was not included in the old template, the coding issue did not affect these numbers which should have stayed the same. However, we did identify that for a small number of participants, follow-up information was included prematurely. For example, data for a 4-week follow-up was conducted after 3 weeks of employment. For consistency, we have removed these cases. This is why the new numbers for the 4-week follow-up are now lower than the old numbers.

By removing participants who have not been in employment for long enough to reach the follow-up point from the denominator, the employment progression rates are consistently higher. This more accurately reflects the real-world situation than was previously the case.

Example: Old methodology versus new methodology In the old methodology, the 13-week follow-up employment progression rate was calculated as follows: (8,130 𝑝𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑛𝑡𝑠 𝑖𝑛 𝑒𝑚𝑝𝑙𝑜𝑦𝑚𝑒𝑛𝑡 𝑎𝑡 13 𝑤𝑒𝑒𝑘 𝑓𝑜𝑙𝑙𝑜𝑤 𝑢𝑝 ÷ 20,743 𝑝𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑛𝑡𝑠 𝑤ℎ𝑜 𝑒𝑛𝑡𝑒𝑟𝑒𝑑 𝑒𝑚𝑝𝑙𝑜𝑦𝑚𝑒𝑛𝑡) × 100% = 39.2%   In the new methodology, the 13-week follow-up employment progression rate will be calculated as follows (includes corrected coding issue): (11,247 𝑝𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑛𝑡𝑠 𝑖𝑛 𝑒𝑚𝑝𝑙𝑜𝑦𝑚𝑒𝑛𝑡 𝑎𝑡 13 𝑤𝑒𝑒𝑘 𝑓𝑜𝑙𝑙𝑜𝑤 𝑢𝑝 ÷ 19,524  𝑝𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑛𝑡𝑠 𝑤ℎ𝑜 𝑒𝑛𝑡𝑒𝑟𝑒𝑑 𝑒𝑚𝑝𝑙𝑜𝑦𝑚𝑒𝑛𝑡 𝑜𝑣𝑒𝑟 13 𝑤𝑒𝑒𝑘𝑠 𝑎𝑔𝑜)  × 100% = 57.6%

As noted earlier, progression statistics are currently released as a total over the full period of No One Left Behind and there are no year/quarter breakdowns. From May 2025, we will break the statistics down by year and quarter, as well as providing the overall total. This will allow changes over time to be observed. It should be noted however that due to the change in methodology, there will be no data available in the most recent quarters for the latter follow-up points because no participant will have been employed long enough to have reached these points. We will also provide additional data on the average number of days between starting support and entering employment.

Please note, we will publish data using both the old and new methodology in May 2025 to ensure users can clearly see the impact of this change, however, we will release the old data as an “archive” tab that will be phased out.   

5. What are the changes proposed to take place beyond May 2025?

The other potential improvements to our methodology that we set out in Section 3 of this paper cannot be implemented with immediate effect. This includes further breakdowns of the follow-up responses such as counting how many participants that entered employment have gone on to other positive destinations, could not be contacted, have confirmed they are no longer in employment etc to provide further context to the statistics. We are also looking at additional outcomes and re-engagement, and the potential release of data related to pay, hours, contracts, and employment sector. However, as this work requires extensive collaboration with data suppliers and further analysis, we are not able to put a timeline on this currently.

This paper has focused on entering employment as a positive destination. As noted previously, however, there are other positive outcomes that participants can achieve which are also reported: further/higher education, gaining a qualification, re-engaging with school, starting work experience or volunteering. 

While follow-up information is also recorded for some of these outcomes, this has not been published to date. We will continue to release overall numbers for these other outcomes in May 2025 and onwards. We will also look to develop these further over time where possible, for example by considering breakdowns by year and quarter and where collected, releasing follow-up information. This will require a similar deep dive exercise as carried out here for employment so will take time to implement. 

We are working with the users of our data to prioritise the statistical improvement work outlined in this paper based on most urgent need. We will implement additional improvements when possible and keep users updated on our progress through our quarterly publications. 

Another option which offers great potential in addressing data quality issues and challenges in relying on participants responding to follow-ups, is the potential to link No One Left Behind data with other sources that already have employment information such as HMRC data. Data linkage work is complicated, however we are scoping out with relevant stakeholders the feasibility of this.

6. Conclusion

The statistics for No One Left Behind are official statistics in development which means they are undergoing continual development in line with the standards of trustworthiness, quality, and value in the Code of Practice for Statistics. As part of this, we are continuously reviewing our methodology to ensure our statistics meet user needs and accurately reflect the work of partners.

This deep dive into the progression methodology for employment outcomes has provided important insights into the strengths, challenges and opportunities of this data, allowing us to develop an improvement plan that ensures our data is accurate and valuable to users. This exercise also identified some data quality issues that we have been able to correct immediately such as fixing the issue on recording from the old template. We have also implemented improvements to our guidance documents, and we’ve started scoping work with data suppliers on further improvements to this data.

We will look to implement the proposed changes in this paper over the short, medium, and long term, working in partnership with those collecting, providing and using this data.

Table 4: Summary of proposed improvements and timelines

When?	What?
May 2025	Break down progression statistics by year and quarter and calculate average time taken for participants to reach employment.
May 2025	Calculate the number of participants who have had enough time to reach the follow-up point and use that as the denominator to calculate job sustainment percentages.
Beyond May 2025	Include all follow-up responses in our analysis.
Beyond May 2025	Consider additional outcomes, including participants who have achieved employment as an outcome more than once.
Beyond May 2025	Include information on contract type, number of hours worked per week, rate of pay and employment sector in our analysis.