The Scottish Government’s Quality Assessment of the Office for National Statistics Labour Force Survey and Annual Population Survey data for Scotland

Annex 1: Detailed Quality Assessment Results for Labour Market Trends

The Labour Market Trends publication is sourced from the ONS Labour Force Survey (LFS) and contains headline estimates of employment, unemployment, and economic inactivity for Scotland. These headline estimates are primarily compared with estimates for the UK but also with headline estimates for the other three countries of the UK – England, Wales, and Northern Ireland. In addition, headline estimates by sex are included for Scotland and the UK. It should be noted that the estimates included in the Labour Market Trends publication are seasonally adjusted whereas all analysis in this paper is based on non-seasonally adjusted estimates.

Sample Sizes

When considering the change in sample size for Scotland between Jul-Sep 2019 and Jul-Sep 2024 from the LFS:

• the achieved sample size for all estimates has dropped considerably
• the average decrease was -41.2%
• the largest decrease was for “unemployed women” which fell by -61.9%
• the smallest decrease was for “economically inactive men” which fell by -26.8%

Figure A1: Since 2019, sample sizes for people in Scotland have decreased for all headline labour market indicators and for both sexes

Labour Force Survey estimates of percentage change in sample size between Jul-Sep 2019 and Jul-Sep 2024 for headline labour market indicators by sex

Coefficient of Variation

When considering the change in the coefficients of variation (CV) between Jul-Sep 2019 and Jul-Sep 2024 for the Scottish LFS estimates:

• all CVs have increased
• the average increase was 1.58
• the largest increase was “unemployed women” which rose by 5.47 from 12.04 to 17.51
• the smallest increase was “employed people” which rose by 0.25 from 0.89 to 1.14

Although a CV of 20 is the threshold for current data quality rules, the three unemployment estimates included in our publications (people, men and women) were above 10, indicating a level of volatility in the data that causes concern.

Figure A2: Since 2019, coefficients of variation for the labour market estimates in Scotland have increased for all indicators assessed

Labour Force Survey estimates of coefficients of variation for headline labour market indicators by sex, Jul-Sep 2019 and Jul-Sep 2024

Current Data Quality Rules

The labour market estimates for Scotland reported in the Labour Market Trends publication were considered “robust” for the Jul-Sep 2024 LFS data. The exception was the estimate for “unemployed women” in Scotland which was considered “less robust”.

Table A1: LFS estimates used for Scotland in the Labour Market Trends publication and their status in Jul-Sep 2019 and Jul-Sep 2024 (non-seasonally adjusted) based on current data quality rules

Data Confidence Ratings

From the Scottish LFS data reported in the Labour Market Trends publication, three estimates have decreased in confidence rating between Jul-Sep 2019 and Jul-Sep 2024.

Based on Jul-Sep 2024 LFS data for Scotland, we have five estimates with “high confidence”, one estimate with “moderate confidence”, two estimates with “limited confidence”, and one estimate with “low” confidence.

Table A2: LFS estimates used for Scotland in the Labour Market Trends publication with confidence ratings in Jul-Sep 2019 and Jul-Sep 2024 (non-seasonally adjusted)

Conclusion of Quality Assessment for the Labour Market Trends publication for Scotland

Overall, the LFS data for Scotland has seen a sharp decrease in sample sizes since the pandemic which has led to increased volatility (CVs) in the estimates.

This is particularly evident in the unemployment data for Scotland where sample sizes have reduced by up to 62% and CVs are between 10 and 20.

Given the “low confidence” and “limited confidence” we have in the key unemployment estimates required to provide a full and robust picture of the labour market in Scotland, it is increasingly difficult for our analysts to provide the data needed by our users.

Due to the particularly high profile nature of these statistics we took the decision that a “less robust” estimate, while still robust enough to publish, was likely to be used incorrectly in some instances. To avoid misleading users, we stopped providing estimates by sex in our Labour Market Trends publication until such time as improvements are seen in the quality of the data.

All other figures are considered “robust” under current data quality rules. However, due to the continuing small sample sizes and increased CVs for even the overall Scotland level data, the Scottish Government, along with ONS, continue to advise caution when using the employment, unemployment, and economically inactive estimates for Scotland from the LFS. These figures should instead be considered alongside the suite of labour market data included in the monthly Labour Market Trends publication such as HMRC Pay As You Earn (PAYE) Real Time Information (RTI), and the Claimant Count data.