Scotland's Devolved Employment Services: Review of Employment Progression Methodology

Appendix B

Data quality issues surrounding the follow-up responses

As part of the deep dive into the data, we closely examined the follow-up responses and surrounding data quality issues. This analysis considered two sets of participants: (1) those who had their entire progression journey recorded within the new template, and (2) those who had their entire progression journey recorded within the old template.

Response options returned in the new template

Our analysis concerning those who had their entire progression journey recorded within the new template covers 6,978 participants who started from October 2022 and had an employment start date up to 30th June 2024 (the latest date covered in the October 2024 publication). 

We found that most follow-ups that were not recorded as “In employment” were either left blank or returned as “unknown”. For example, when removing those recorded as still “in employment” at each follow-up and removing those that have not been in employment for long enough to reach particular follow-ups, blanks and “unknowns” accounted for an average of 90% of follow-up responses. In contrast, both “Not in employment” and “No engagement” is returned for an average of 3% of follow-up responses each. See table 1 below for breakdowns. 

This table also gives an idea of how many participants were included in the denominator for calculating still in employment at each follow-up despite not having been employed long enough to reach this stage (“Not enough time passed” row).

Table B1: Response options returned for follow-ups in the new data template

Guidance states follow-up attempts should continue even if participants state they are no longer in employment or don’t respond to earlier follow-up points. The data shows this may not always be occurring as there are examples of follow-up points after a “not in employment” response being blank.

There is local variation in the use of follow-up responses. Some data suppliers have zero use of the “Not in employment” option, while others have higher use. The conclusion that no participant entering employment then falls into unemployment, especially as this includes areas with a large percentage of participants, seems unlikely and points to issues with leaving follow-ups as blank if the participant is no longer in employment.

Almost half of data suppliers have zero participants with the “No engagement” option which seems unlikely to be an accurate reflection given known difficulties with reaching participants for follow-ups. It was noted however that while an option in the data template, this was not included in the data guidance which may have impacted its use. This has since been corrected.

Taken together, the data suggests that if contact is made with a participant and they are still in employment, this is likely to be returned. However, if the participant is not in employment or cannot be contacted, some data suppliers may leave this blank rather than use the other available options. This presents significant challenges for providing additional breakdowns on lack of engagement or accurately reporting how many fall out of employment which are both important information for providing a fuller picture on what happens to No One Left Behind participants after entering employment. We are working with data suppliers on this and have issued a survey to better understand the use of these other follow-up options which will inform our work going forward to improve this.

Response options returned in the old template

The table below shows the follow-up breakdowns for the participants whose progression journeys were recorded within the old template. Please note that the old template did not include a 4-week follow-up check, nor did it include a “No engagement” option.

The breakdowns shown below suggests that the old template data has similar quality issues compared to the new template. While “Entered employment” values seem to have been recorded consistently, the number of unknowns/blanks is high and “Not in employment” values are often not recorded at all.

Table B2: Response options returned for follow-ups in old data template