The Scottish Government’s Quality Assessment of the Office for National Statistics Labour Force Survey and Annual Population Survey data for Scotland

1 Key Points

The problem

• in recent years there has been increased difficulties in getting people to respond to surveys and other data collections. These problems are affecting most surveys across the UK
• the Office for National Statistics (ONS) Labour Force Survey (LFS) and Annual Population Survey (APS) supply the labour market statistics for Scotland. These surveys have experienced significant reductions in response rates over recent years
• as a result, both surveys have seen increased volatility and uncertainty in the estimates. This has resulted in wider confidence intervals (CIs) around the estimates and larger coefficients of variation (CV) associated with each estimate
• it is increasingly difficult to confidently report on labour market statistics for Scotland based on the time periods assessed

What we did

• we took snapshots in time (one pre-pandemic and one post-pandemic) for each of the 241 key estimates we regularly publish from the LFS and APS datasets
• we adopted a 2-stage approach to assessing the quality of the labour market statistics for Scotland:
  • first, we assessed estimates against our current data quality rules, which take account of sample sizes and the coefficient of variation for each estimate, to determine whether estimates are “robust”, “less robust”, or “not robust”
  • second, we applied data confidence ratings to the estimates to inform users of the level of confidence we have in each of the estimates. Estimates may pass the data quality rules but we may still have a low degree of confidence in how they can be used

Data Quality Results

Of the 12 LFS estimates assessed:

• the average reduction in sample size between 2019 and 2023/2024 was -45.5%
• the average CV for the 2023/2024 LFS estimates assessed was 6.36, higher than in 2019 (4.65)
• eleven were considered “robust” under current data quality rules for the 2023/2024 data
• one was considered “less robust” for the same time period

Of the 229 APS estimates assessed:

• the average reduction in sample size between 2019/2020 and 2023 across all estimates (excluding any with increasing sample) was -60.5%
• the average CV for all 2023 APS estimates assessed was 10.40, higher than in 2019 (6.43)
• there were 181 estimates considered “robust” under the current data quality rules for the 2023 data
• 31 were considered “less robust” for the same time period
• 17 estimates were “not robust” for the same time period

Table 1: Labour Force Survey (LFS) and Annual Population Survey (APS) estimates by current data quality rules in 2023/2024 (LFS) and 2023 (APS)

Data Confidence Results

• our confidence in 122 (50.6%) estimates has reduced over the time period assessed
• For the latest time period assessed (2023/2024), there was:
  •  “high confidence” in 90 (37.3%) of the estimates indicating accurate conclusions can be drawn from these estimates as they provide a good reflection of reality
  • “moderate confidence” in 52 (21.6%) estimates indicating the data is suitable, but incomplete and limitations should be understood before use
  • “limited confidence” in 52 (21.6%) estimates indicating these estimates provide a restricted view on reality and caution is required whilst using these estimates and these should be viewed alongside the suite of other indicators
  • “no confidence” in 27 (11.2%) estimates and “low confidence” in 19 (7.9%) estimates indicating these estimates are not fit for publication as principal measures of labour market information for Scotland

Table 2: Labour Force Survey (LFS) and Annual Population Survey (APS) estimates by data confidence ratings in 2023/2024 (LFS) and 2023 (APS)

Conclusion

Estimates from the LFS and APS for Scotland are showing clear signs of increased volatility and reduced quality as a result of, but not exclusively due to, continued effects from the COVID-19 pandemic.

The ONS have made progress in working to mitigate these issues. However, we feel it important to be transparent and communicate to users our ongoing concern with the quality of the estimates obtained from these surveys and the impact this has on statistics for Scotland.

Based on the data confidence ratings, we found that 37.3% of Scottish LFS and APS estimates assessed were considered to have “high confidence” at present. However, 40.7% of Scottish estimates were considered to have “no confidence”, “low confidence”, or “limited confidence” indicating some are not fit for purpose, and others need to be considered in the context of the wider suite of labour market data.

The “low confidence” and “no confidence” ratings for many of the key estimates make it increasingly difficult for our analysts to provide the data needed by our users.

We will continue to assess the quality of the LFS and APS data we receive from ONS using the current data quality rules to determine what we can and cannot continue to publish until we have completed our full assessment of the data. We are engaging with users on the availability and usefulness of estimates, and considering all ad-hoc data requests on a case by case basis using these rules and guidelines. We are also assessing the confidence intervals around estimates to determine if the data is of high enough quality to provide meaningful results.

We continue to consult with ONS on the quality of the LFS and APS data and work is ongoing to explore options for improving the Scottish Labour Market data to ensure we can meet user needs as effectively as possible