The Scottish Government’s Quality Assessment of the Office for National Statistics Labour Force Survey and Annual Population Survey data for Scotland

5 Conclusion

This paper completes phase one of the Scottish Government’s quality assessment of the Office for National Statistics (ONS) Labour Force Survey (LFS) and Annual Population Survey (APS), focussing on the Scottish statistics derived from these. We have set out the current status for the key labour market statistics for Scotland, indicating which estimates pass the current data quality rules and which estimates pass the more nuanced data confidence rating metrics. With the data confidence rating we can indicate which estimates we have determined we can continue to publish and which estimates we do not have confidence in.

Estimates from the LFS and APS are showing clear signs of increased volatility and reduced quality as a result of, but not exclusively due to, continued effects from the COVID-19 pandemic.

When applying the current data quality rules, 192 of the 241 estimates we publish or supply the data for meet the current data quality standard of “robust” and could continue to be published. However, 49 estimates do not meet the current “robust” data quality rating.

Of the 12 LFS estimates we assessed, 11 were considered “robust” for the 2023/2024 period and one was considered “less robust”.

Of the 229 APS estimates we assessed, 181 were considered “robust” for the 2023 data. Of the remaining estimates, 48 did not meet the current data quality standard of “robust”, with 31 considered “less robust” and 17 estimates considered “not robust” for the 2023 period.

However, the current data quality rules do not capture the nuances in the data that determine whether a statistic is fit for publication, as a “robust” estimate may only meet the minimum requirements in terms of sample size. We therefore also applied data confidence ratings to the estimates, to clearly inform users of the level of confidence we have in the estimates (as estimates might pass the data quality rules but still have a low degree of confidence in how they can be used). This is the case for any data that has been assessed as “no confidence”, “low confidence”, or “limited confidence” in our data confidence ratings.

In doing so we found we have less confidence in 122 (50.6%) estimates in 2023/2024 compared to 2019.

Specifically, we have “no confidence” in 27 estimates (11.2%) and “low confidence” in 19 estimates (7.6%) indicating that these estimates are not fit for purpose as principal measures of labour market information for Scotland.

We have “limited confidence” in 52 estimates (21.6%) indicating these estimates provide a restricted view on reality and caution is required whilst using these estimates and these should be viewed alongside the suite of other indicators.

We have “moderate confidence” in 52 estimates (21.6%) indicating the data is suitable, but incomplete and limitations should be understood before use. Finally, out of the 241 estimates we assessed, we have “high confidence” in 90 (37.3%) of the LFS or APS estimates indicating these are the only estimates that provide a good reflection of reality and from which we can draw accurate conclusions.

Overall, we remain concerned about the falling sample sizes across the periods assessed and the increased coefficients of variation which make the Scottish labour market data harder to interpret for us and for our users.

It is also concerning that many headline labour market estimates, which are high profile and are regularly quoted in the media, are based on a very low number of responses. The Office for National Statistics have reassured the Scottish Government that their weighting methodology is appropriate for sample sizes as low as 20 and will still be representative, however the increasing volatility and uncertainty arising from the smaller samples still pose a challenge to accurately interpretating the data.

Given the “low confidence” and “no confidence” rating for many of the key estimates required to provide a full and robust picture of the labour market in Scotland, it is increasingly difficult for Scottish Government analysts to provide the data needed by our users. This is an issue particularly for the unemployment data in both the LFS and APS estimates for Scotland. For example, if the unemployment estimates for women do not meet the quality rules to publish, we cannot publish unemployment estimates for men either due to users being able to calculate the women’s unemployment figures by subtracting the men’s unemployment figures from the total (people) unemployment figures for Scotland. Therefore, we would not be able to publish the breakdown of unemployment by sex.

Due to the high profile nature of the labour market data, we will continue to remind users that any short term changes in the statistics should not be used to infer actual changes unless these are statistically significant. We also include notes in our outputs to advise users that short term changes are likely to be influenced by the overall volatility of the data rather than real world changes in the labour market.

We will continue to work with ONS and others to explore all options available to improve the vital Scottish Labour Market data.